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FOREWORD 


The  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences  (ARI)  performs  research  and  development  in 
areas  that  include  training  performance  measurement  and 
assessment  (PMA)  throughout  the  Army.  Of  special  interest  is 
how  the  data  that  are  generated  as  a  result  of  training  PMA 
are  used  to  improve  the  training  that  takes  place  in  the 
Army. 

In  order  to  perform  detailed  research  for  PMA  in  tech¬ 
nical  training,  it  was  first  necessary  to  thoroughly  examine 
the  existing  literature  base.  By  adequately  reviewing  past 
and  current  attempts  at  measuring  and  assessing  training  per¬ 
formance,  we  can  better  support  research  efforts  to  improve 
the  Army's  PMA  system.  Such  improvement  is  necessary  to 
address  the  Army  Science  Board's  Summer  Study.  The  study 
concluded  that  the  Army's  system  of  training  PMA  is  often  not 
providing  adequate  feedback  to  the  developers  of  training 
systems. 

This  project  was  performed  under  the  research  task 
called  "Methods  for  Evaluating  Training  System  Effective¬ 
ness."  The  project  supports  the  Orlando  Field  Unit's  mission 
of  developing  methods  for  the  optimization  of  simulation- 
based  training  systems,  training  performance  measurement 
being  a  key  part  of  any  training  system.  It  supports  the 
Training  Research  Laboratory's  research  program  by  examining 
the  current  conduct  of  training  performance  measurement  and 
assessment  activities  in  Army  schools  and  operational  units. 
There  were  two  sponsors  of  this  research.  One  sponsor  was 
the  Army's  Project  Manager  for  Training  Devices  (PM  TRADE) 
under  a  Memorandum  of  Understanding  dated  18  May  1983  and 
entitled  "Establishment  of  Technical  Coordination  between  the 
ARI  and  PM  TRADE."  The  other  sponsor  was  the  Department  of 
Defense's  Training  and  Performance  Data  Center  (TPDC)  under  a 
Memorandum  of  Agreement  entitled  "Army  Research  Institute 
Coordination  with  TPDC"  and  dated  24  April  1985.  The  Com¬ 
manding  Officer  of  PM  TRADE  and  the  Director  of  TPDC  were 
briefed  on  the  project  results  in  September  of  1987.  The 
proponents  (sponsors)  expect  to  make  use  of  the  project's 
findings  in  the  future  design  of  their  training  system  (PM 
TRADE)  and  in  the  future  collection  of  data  (TPDC) . 
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A  REVIEW  AND  ANNOTATED  BIBLIOGRAPHY  OF  TRAINING  PERFORMANCE 
MEASUREMENT  AND  ASSESSMENT  LITERATURE 


EXECUTIVE  SUMMARY 


Requirement : 

The  objective  of  the  report  is  to  provide  a  foundation 
for  technical  training  performance  measurement  and  assessment 
(PMA)  research  within  the  military.  This  review  was  per¬ 
formed  to  support  a  specific  research  project,  the  results  of 
which  appear  in  a  separate  report  entitled  "Measuring  and 
Assessing  Technical  Training  Performance  in  the  Army's 
Schools  and  Units:  A  Survey  of  Current  Methods." 


Procedure : 

A  literature  review  was  made  of  recent  and  current 
research  efforts  that  address  training  PMA  concerns.  To 
initiate  the  search,  computer-assisted  and  manual  searches 
were  employed.  The  following  data  bases  were  accessed: 
Defense  Technical  Information  Center;  Educational  Resources 
Information  Center;  PSYCH  INFO,  American  Psychological  As¬ 
sociation;  and  the  Conference  Papers  Index.  In  addition, 
listings  of  Army  Research  Institute  Publications  dating  from 
1940  to  1986  were  reviewed  for  pertinence.  Of  the  documents 
reviewed,  173  are  presented  in  annotated  format  in  Appendix  A 
of  the  report. 


Findings : 

A  review  of  PMA  literature  has  revealed  that  the 
services,  including  the  Army,  have  not  succeeded  in  devel¬ 
oping  an  integrated  system  for  measuring  and  assessing  train¬ 
ing  performance.  Several  specific  problems  were  indicated  by 
the  review.  There  is  still  an  over  reliance  on  subjective 
measures  of  performance  and  a  shortage  of  valid,  reliable 
quantitative  performance  measures  of  training  objectives, 
training  strategies,  and  training  effectiveness.  Research 
efforts  should  be  pursued  to  resolve  these  problems.  Re¬ 
searchers  should  investigate  means  for  developing  more 
empirical  data,  better  analytic  methods,  and  standardized 
measurement.  The  military  should  assess  the  accuracy  of 
subjective  feedback  and  develop  better  methods  for  managing 
and  utilizing  feedback  information.  Increased  emphasis 
should  be  placed  on  the  application  of  learning  principles 
such  as  knowledge  of  results  and  retention  of  learning  in 
designing  PMA  systems. 


vii 


Utilization  of  Findings: 

In  order  to  adequately  address  the  critical  findings  of 
the  1985  Army  Science  Board's  Summer  Study  concerning  PMA  in 
the  Army,  it  will  be  necessary  to  conduct  research  to  iden¬ 
tify  PMA  solutions.  This  report  lays  a  literature-based 
foundation  for  such  a  program.  In  addition,  the  developers 
of  PMA  systems  should  be  able  to  glean  ideas  for  improving 
their  designs  from  this  report. 
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A  REVIEW  AND  ANNOTATED  BIBLIOGRAPHY  OF  TRAINING 
PERFORMANCE  MEASUREMENT  AND  ASSESSMENT  LITERATURE 


OVERVIEW 


Operational  Problem 

In  order  for  the  Army's  training  system  to  be  as 
responsive  to  the  needs  of  the  Army  as  possible,  it  is 
imperative  that  there  be  a  means  to  measure  the  performance 
of  soldiers  and  units  after  training.  Performance 
measurement  has  been  defined  as  the  scoring  of  trainee 
proficiency  either  subjectively  (e.g.,  instructor  opinion)  or 
objectively  (e.g.,  automatic  computer  measurement).  (MIL- 
HDBK-220B. )  Performance  assessment  involves  the  synthesis  of 
all  performance  measurement  information  to  assess  trainee 
performance.  (MIL-HDBK-220B. )  This  training  performance 
measurement  and  assessment  has  a  very  important 
goal — it  provides  the  Army's  chain  of  command  with 
information  as  to  the  state  of  readiness  of  the  Army,  as  well 
as  indications  as  to  possible  causes  of  performance  that  is 
less  than  acceptable.  Recent  studies  such  as  that  of  the 
1982  Defense  Science  Board  (DSB)  and  the  Summer  Study  of  the 
1985  Army  Science  Board  (ASB)  suggest  that  there  is  a  lack  of 
valid  and  reliable  training  performance  information  to  help 
guide  these  decisions.  This  is  further  supported  by  research 
conducted  in  the  area  of  training  PMA. 

Resgarsli  Qfrtectiygg 

The  objective  of  this  literature  review  was  to  lay  a 
foundation  for  programmatic  training  performance  measurement 
and  assessment  (PMA)  research.  The  review  directly  supported 
a  PMA  research  project,  the  results  of  which  appear  in 
"Measuring  and  Assessing  Technical  Training  Performance  in 
the  Army'o  Schools  and  Units:  A  survey  of  Current  Methods." 

SSfiRS 


In  order  to  accomplish  the  research  objectives,  the  Army 
Research  Institute  (ARI)  Orlando  Field  Unit  initiated  a  study 
effort  to  1)  determine  how  the  Army  measures  performance, 

2)  determine  how  the  resulting  data  and  information  are  used 
to  improve  Army  training,  3)  investigate  the  cost 
effectiveness  of  the  methodologies  used  to  measure 
performance,  and  4)  assess  the  contribution  that  training 
devices  and  simulators  make  to  the  effectiveness  of  the 
training  PMA  system.  The  literature  review  to  support  tasks 
one  and  two  is  contained  in  this  report. 
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BACKGROUND 


The  Defense  Science  Board's  Summer  Study  of  1982,  as 
well  as  the  Army  Science  Board's  1985  Summer  Study,  were 
critical  of  the  methodologies  used  by  the  Armed  Services  (in 
the  case  of  the  DSB)  and  the  Army  (in  the  case  of  the  ASB) . 
Specific  criticisms  pointed  to  the  lack  of  objective 
standards  to  measure  human  performance  and  the  lack  of 
quantifiable  measures  of  performance  as  two  of  the  most 
serious  of  the  shortcomings.  Other  studies  (to  be  described 
later  in  this  report)  that  looked  at  training  PMA  in  both  the 
individual  and  collective  training  environments  also  found 
fault  with  some  of  the  procedures  used  within  the  Army  to 
measure  performance  in  order  to  assess  training.  Many  of 
these  research  efforts  produced  findings  that  echoed  the  DSB 
and  ASB  Summer  Studies.  A  recurrent  theme  in  many  of  these 
studies  has  been  the  questionable  validity  and  reliability  of 
the  training  PMA  methodologies  used  by  the  Army.  The  Army 
Research  Institute  (ARI)  designed  the  current  effort  to 
explore  more  fully  the  problems  that  had  been  cited,  and  to 
provide  a  basis  for  suggesting  ways  by  which  training  PMA  in 
the  Army  might  be  improved. 

Me.tti949logy 

As  a  first  step  in  addressing  the  concerns  cited  above, 
a  literature  search  was  conducted.  To  initiate  the 
literature  search,  computer-assisted  and  manual  searches  were 
employed.  The  computer-assisted  literature  search  accessed 
dialog  across  the  following  data  bases:  Educational 
Resources  Information  Center  (ERIC) ;  PSYCH  INFO,  American 
Psychological  Association;  and  the  Conference  Papers  Index. 
The  search  was  run  in  the  data  base  on  key  words  to  include 
"performance  measurement,"  "training  effectiveness," 

"training  measurement,"  "decision  making,"  etc.  Results  from 
this  initial  search  suggested  modifications  in  the  number  of 
data  bases  to  access.  Ninety-three  citations  were  obtained 
and  only  31  citations  were  selected  for  review. 

Concomitantly,  the  Defense  Technical  Information  Center 
(DTIC)  was  accessed.  This  search  proved  to  be  more  relevant; 
out  of  797  citations,  65  were  selected  for  further  review. 
Additionally,  listings  of  ARI  publications  dating  from 
1940-1986  were  reviewed  for  pertinence.  Out  of  3,500,  173 
were  selected  for  analysis.  As  these  reports  were  reviewed, 
pertinent  data  (to  include  the  title,  author,  publication 
data,  summary,  relevant  points,  and  relevant  subtask)  were 
gathered  and  entered  into  the  data  base.  An  annotated 
bibliography  containing  these  data  is  presented  in 
Appendix  A. 
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Although  many  sources  were  accessed ,  the  source  that 
proved  to  be  the  most  comprehensive  was  DTIC,  the  DoD's 
repository  for  technical  reports  from  Government,  academia, 
and  industry.  Although  there  apparently  has  been  a  great 
deal  of  discussion  of  training  PMA  in  all  sectors,  the 
research  which  is  most  applicable  to  the  current  effort  seems 
to  have  been  conducted  in  the  military  arena.  As  clearly 
stated  by  Vreuls  and  Woolridge  (1977),  "In  order  to  increase 
efficiency  and  maintain  (or  improve)  current  system 
effectiveness,  we  must  improve  measurement.  Responsiveness 
to  this  need  can  be  seen  in  the  ongoing  training  performance 
measurement  programs  in  the  Navy,  Air  Force,  and  Army."  In 
the  Army,  an  increased  interest  in  its  training  performance 
measurement  system  is  evident.  Witness  the  1985  Army  Science 
Board  Summer  Study  (Army  Science  Board,  1985)  which  concluded 
that  the  Army  has  several  needs: 

•  ' Quantitative '  measures  (of  performance)  relating 
to  training  objectives,  training  strategies  and 
training  effectiveness. 

e  'Quantifiable'  tasks  whose  successful  performance 
to  standards  leads  to  mission  accomplishment. 

•  Evolution  of  measurement  programs  to  a 
'quantifiable'  basis. 

•  Identification  of  task  data  needed  to  measure 
effectiveness  of  training. 

•  Knowledge  of  skills  retention/learning  rates  to 
support  unit  sustainment  training. 

The  Air  Force  has  expended  considerable  effort  toward 
the  development  of  aircrew  training  PMA  systems  (Waag  and 
Knoop,  1977) ,  particularly  through  research  conducted  by  the 
Air  Force  Human  Resources  Laboratory  (AFHRL) .  The  Navy  has 
proposed  the  use  of  an  integrated  approach  to  training  PMA 
(Pettit  and  Magruder,  1985)  and  has  stressed  the  need  for 
quantitative  and  qualitative  performance  measures  which  are 
systematically  developed. 

The  state-of-the-art  in  training  PMA  in  the  Army  is  most 
germane  to  this  report.  Thus,  the  first  part  of  this  section 
addresses  the  following: 

e  How  does  the  Army  measure  performance  of  its 
soldiers? 

e  How  effective  are  these  methodologies? 


•  What  research  has  already  been  done  to  investigate 
the  methodologies  and  their  effectiveness? 

The  literature  review  also  provided  an  insight  into 
ongoing  training  PMA  programs  (in  the  Army,  Navy,  and  Air 
Force)  that  postulate  solutions  to  the  training  PMA  problem. 
In  view  of  this,  the  second  part  of  this  section  addresses 
the  following  questions: 

•  What  recommendations  are  made  to  improve  training 
PMA? 

•  What  ongoing  training  PMA  programs  are  most 
relevant  to  this  study? 

The  various  literature  that  was  reviewed  as  part  of  the 
literature  search  is  enclosed  at  Appendix  A  in  annotated 
bibliography  form. 


PRESENT  PERFORMANCE  MEASUREMENT  SYSTEM 
Measurement  in  Institutions 

Results  of  the  literature  search  and  review  provided  no 
reports  specifically  addressing  how  individual  and  collective 
performance  is  currently  being  measured  in  the  TRADOC  schools 
and  centers.  However,  some  of  the  literature  provided  by  the 
Training  Effectiveness  Analysis  (TEA)  Division  in  TRASANA  is 
relevant  to  this  issue.  The  TEA  Division  in  TRASANA  is 
responsible  for  managing  the  TEA  process  in  support  of  the 
TRADOC  schools  and  centers.  To  date,  the  division  has 
conducted  numerous  TEA  studies,  covering  a  wide  range  of 
areas:  training,  training  devices,  soldier  hardware 
interface,  and  MOS  selection  criteria  (Miller  and  Southard, 
1982) .  Table  1  presents  a  list  of  studies  with  the  type  of 
training  and  MOS  Number  for  institutional  training,  as 
presented  by  Miller  and  Southard. 


4 


Table  1 

Studies  on  Institutional  Training 


STOPX 

Basic  Rifle  Marksmanship 
Cost  and  Training 
Effectiveness  Analysis  (CTEA) 
REDEYE  Weapon  Systems 
Training  Effectiveness  Study 
REDEYE  Weapon  Systems  Army 
Training  StudY  (ARTS) 

VULCAN  Weapons  System  Training 
Subsystem  Effectiveness 
Analysis 

CHAPARRAL/REDEYE  Training 
System  Effectiveness  Analysis 
Training  Attrition  Problem , 
Institute  for  Military 
Assistance  Training  System 
in  Effectiveness  Analysis 


TYPE  OF  INSTITUTIONAL  MOS 

TRAINING _  ML. 

Basic  Combat  Training  N/A 

Advanced  Individual  16P 

Training 

Advanced  Individual  16P 

Training 

Advanced  Individual  16P 

Training 

Advanced  Individual  16P 

Training 

Special  Qualification  15E 

Course  Advanced 
Individual  Training 


The  literature  review  was  not  intended  to  reveal 
specific  training  performance  measures  (e.g. ,  tasks, 
simulated  performance,  actual  performance)  used  in  the  units 
and  institutions.  However,  specific  measures  were  identified 
in  a  detailed  survey  of  six  MOS.  That  survey,  which  was 
part  of  this  research  project,  is  reported  in  a  separate 
report  "Measuring  and  Assessing  Technical  Training 
Performance  in  the  Army's  Schools  and  Units:  A  Survey  of 
Current  Methods."  This  literature  review  was  aimed  more  at 
the  difficulties  which  the  military  has  had  in  establishing 
whole  PMA  systems. 

Some  of  the  relevant  problems  encountered  across  most 
TEA  studies  related  to  training  subsystems  at  the 
institutions  are:  (1)  insufficient  hands-on  training,  (2) 
insufficient  testing  within  a  POI,  (3)  inconsistencies 
between  POI's  and  Soldier  Manual  (SM)  task  lists,  (4)  too 
much  emphasis  on  "irrelevant  training,"  (5)  failure  to  adhere 
to  the  requirements  of  a  training  program,  and  (6)  standards 
and  objectives  of  various  tests  which  are  inconsistent  and 
sometimes  conflicting  with  each  other. 

At  the  institutional  level,  training  is  conducted  on 
tasks  contained  in  the  Soldiers  Manual  (SM)  prepared  for  the 
specific  MOS  and  skill  level.  By  Army  doctrine,  the  SM  is 
the  central  document  around  which  skill  training  revolves. 
However,  the  SM  has  not  been  as  effective  as  it  could  be.  As 
stated  by  Simpson  et  al.  (1984),  SMs  have  not  been  entirely 
effective  for  a  number  of  reasons: 
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•  The  list  of  tasks  they  contain  is  incomplete. 


•  Many  tasks  are  left  out  because  they  are  not  critical 
for  the  MOS. 

e  Even  with  their  omissions,  SMs  contain  an  enormous 
number  of  tasks,  and  these  are  listed  without 
priorities  assigned  to  them. 

Measurement  in  Units 

The  Army's  policy  changes  of  the  mid  1970s  to  transfer  a 
large  part  of  training  responsibilities  from  formal  schools 
to  operational  units  places  new  burdens  on  unit  personnel 
(Simpson,  McCallum,  McIntyre,  Casey,  and  Fuller,  1984) .  As 
stated  in  TRADOC  Reg  350-7,  "The  Army  training  system  is  an 
integration  of  individual  and  collective  training  conducted 
both  in  the  institutions  and  in  the  units.  Although  TRADOC 
has  the  mission  to  provide  individual  and  collective  training 
for  Army  personnel,  most  of  the  training  is  conducted  in  the 
units."  In  the  units,  the  SM  is  the  central  document  around 
which  skill  training  revolves.  Individual  units  are  required 
to  develop  their  training  plans,  interpret  the  SM,  and  select 
tasks  with  their  respective  priorities.  Feedback  on  how 
individuals  and  units  are  doing  is  obtained  through  routine 
performance  assessments  during  the  course  of  evaluations 
conducted  by  the  unit  (e.g.,  Army  Training  and  Evaluation 
Program  (ARTEP) ,  Skill  Qualification  Test,  and  TRADOC  field 
visits.  Unit  needs  are  derived  from  a  delta  between 
performance  required  to  accomplish  their  missions  and  actual 
performance  achieved. 

The  need  for  methods  of  measuring  team  and  unit 
proficiency  and  the  lack  of  knowledge  in  this  area  are  widely 
recognized.  As  stated  by  Knerr,  Root,  and  Word  (1979), 
difficulties  in  measuring  team  performance  are  fundamental 
problems  of  training  evaluation.  Existing  unit  training  PMA 
techniques  depend  largely  on  judgmental  data  and  often  do  not 
evaluate  the  unit's  ability  in  the  field  (Hayes  and  Wallis, 
1979).  This  is  further  reiterated  by  the  1982  Defense 
Science  Board's  Summer  Study  which  noted  that  performance  is 
measured  "subjectively  and  poorly"  (Defense  Science  Board, 
1982)  and  by  the  Army  Science  Board  Summer  Study  which 
concluded  that  the  Army  needs  improvements  in  its  training 
PMA  system  (Army  Science  Board,  1985). 

Army  training  and  evaluation  program.  At  the  unit 
level,  the  literature  review  addresses  training  of 
individuals  and  collective  training.  In  particular,  the 
Skills  Qualification  Test  (SQT)  and  Army  Training  and 
Evaluation  Plan  (ARTEP)  are  addressed.  The  ARTEP  is  utilized 
to  conduct  unit  training  of  collective  skills  and  to  evaluate 
performance  of  these  tasks.  The  ARTEPs  basic  goal  is 
"performance-oriented  training"  (Havron,  Albert,  and 


McCullough,  1978) .  It  encompasses  mission/task  oriented 
training,  concurrent  multiechelon  training  and  evaluation, 
training  to  correct  deficiencies,  and  decentralized  training 
and  evaluation  (Havron  and  Wanschura ,  1979). 

Although  there  is  not  supposed  to  be  any  deviation  from 
the  tasks,  conditions,  and  standards  specified  in  the  ARTEP, 
Havron  et  al.  (1978  and  1979)  found  this  to  be  frequently  the 
case.  A  study  by  Gray,  Clovis,  Gray,  Muller  and  Cunningham 
(1981)  indicates  that  officers  sometimes  modify  the  ARTEP  to 
include  additional  subtasks  required  by  their  unit.  Thus 
ARTEPs  are  frequently  used  to  develop  training  exercises  for 
formal  evaluation,  not  to  develop  training  to  overcome 
weaknesses  identified  in  the  evaluation.  Attachment  to  this 
formal  unit  evaluation  status  has  been  found  to  hinder  the 
usefulness  of  the  ARTEP  as  part  of  the  training  program  since 
emphasis  is  placed  on  passing  the  tests,  rather  than  on 
discovering  and  correcting  deficiencies.  Mirabella  (1978) 
summarizes  this  by  stating  that  the  ARTEP  does  not  adequately 
differentiate  between  evaluation  for  training  diagnosis  and 
evaluation  for  accountability.  Ultimately,  this  degrades 
the  validity  and  reliability  of  the  ARTEP  as  a  training  PMA 
instrument. 

Of  equal  concern  is  the  extent  to  which  ARTEPs  do  not 
provide  for  objective  measures  of  performance.  Numerous 
study  efforts  (Shaket,  Saleh,  and  Freedy,  1981?  Medlin,  1979? 
Allen,  Johnson,  Wheaton,  Knerr,  and  Boycan,  1982?  Hayes  and 
Wallis,  1979?  Havron  and  McFarling,  1979?  and  Havron  and 
Wanschura,  1979)  have  suggested  that  this  is  the  biggest 
shortcoming  of  the  ARTEPs.  Medlin  (1979)  points  out  a  lack 
of  standardized  or  scientific  procedures  for  determining  the 
tasks,  subtasks,  and  standards  in  the  Training  and  Evaluation 
Outlines  (T&EOs)  of  the  ARTEP  manuals.  Furthermore,  the 
field  exercises  are  often  unrealistic  and  do  not  provide 
objective  data  for  the  evaluation  team.  As  a  result,  the 
ARTEP  is  dominated  by  unsystematized  and  unaided  human 
judgments  (Medlin,  1979)  and  a  lack  of  objective  data  to 
determine  terminal  mission  outcome  (Shaket  et  al.,  1981). 

The  use  of  subjective  measurement  techniques  often 
results  in  inaccurate  data  (Burnside,  1982)  which  frequently 
arise  when  human  judgments  are  involved  (e.g.,  halo, 
leniency,  and  central  tendency  errors) .  Subjective 
appraisals  can  provide  accurate  data  if  provisions  are  taken 
to  avoid  these  errors  (e.g.,  adequate  evaluator  training). 
However,  in  the  Army  there  is  a  lack  of  appreciation  for  the 
role  evaluators  can  and  must  play  in  evaluating  performance 
(Havron  et  al.,  1979).  Little  time  is  devoted  to  evaluator 
training  and  when  it  is  provided,  this  training  is  often 
inadequate  (Shaket  et  al.,  1981).  Medlin  and  Thompson  (1980) 
conducted  a  study  to  determine  the  major  dimensions  that 
military  judges  use  in  subjectively  appraising  ARTEP 
performance.  A  complex  multi-dimensional  analysis  of  ratings 
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indicated  that  military  judges  used  only  three  dimensions  and 
that  the  dominant  dimension  was  the  quality  of  overall 
performance.  Thus  their  assessment  was  not  based  on  a 
careful  objective  analysis  of  tasks.  Ultimately,  the  lack  of 
objective  data  and  an  explicit  data  base  make  it  difficult  to 
interpret  the  results  of  an  evaluation,  or  to  compare  the 
same  unit  at  different  times  or  locations. 

While  the  inclusion  of  subjective  standards  has  been  one 
of  the  most  serious  drawbacks  of  the  ARTEPs  published  to 
date,  it  is  not  the  only  problem  identified  by  the 
literature.  Shelnutt,  Smillie,  and  Bercos  (1978)  and  Wheaton 
and  Boycan  (1982)  concluded  that  the  standards  of  performance 
are  often  inaccurate,  too  general,  and  vague.  Moreover,  the 
T&EOs  often  demonstrate  a  lack  of  correspondence  between 
task,  conditions,  and  standards  (Havron  et  al.,  1978). 

Philosophically,  the  ARTEP  is  intended  for  use  in 
training,  evaluation,  and  development  of  training  to  correct 
deficiencies  (Havron  et  al.,  1978).  Most  units  use  the 
results  of  ARTEP  evaluations  to  ascertain  areas  in  which 
additional  training  is  needed.  However,  this  is  done  in  a 
relatively  imprecise  way  due  to  the  structure  of  the  ARTEP. 
Although  ARTEP  results  are  often  not  available,  when 
available  they  are  usually  in  a  general  format  and  do  not 
provide  task-specific  feedback  on  how  to  correct 
deficiencies.  (Burnside,  1981) .  The  T&EOs  are  not  designed 
to  recapture  specific  errors  (Havron  et  al.,  1979). 

Therefore,  it  is  almost  impossible  to  track  specific  errors 
and  integrate  ratings  for  diagnosis.  Burnside  also  points 
out  that  ratings  are  usually  provided  for  each  company  on 
performance  of  a  major  mission  (e.g.,  movement  to  contact), 
but  that  there  is  no  listing  of  specific  individual  or  crew 
level  tasks  which  need  further  training.  As  a  result, 
commanders  at  the  end  of  ARTEP  do  not  get  a  list  of 
deficiencies  or  discriminatory  performance  data  (Hill  and 
Sticht,  1981) .  The  feedback  provided  is  a  go  or  no-go  for 
each  task  evaluated. 

This  feedback  is  too  general  and  as  stated  by  various 
commanders  in  charge  of  ARTEP  evaluation,  "you  have  to  have 
more  detail  than  'sat'  and  'unsat'”  (Gray,  et  al.,  1981). 

The  problem  is  further  aggravated  by  the  inadequate  training 
provided  to  the  evaluators  and  the  lack  of  procedural 
guidelines  and  ratings  across  standard  items,  missions,  and 
unit  echelons  (Shaket  et  al.,  1981).  The  ARTEP  manuals 
provide  little  or  no  guidance  to  evaluators  on  how  to  design 
exercises,  measure  unit  performance,  determine  training 
proficiencies/deficiencies,  or  evaluate  the  observed 
performance  (Medlin,  1979) .  Although  these  problems  are 
evident  in  today's  Army  units,  ARTEP  results  are  still  being 
utilized  for  diagnostic  purposes. 


8 


To  the  extent  that  units  view  the  ARTEP  evaluation  as 
the  culmination  of  training,  as  opposed  to  the  first  step  in 
the  training  program,  the  efficacy  of  the  diagnosis  is 
seriously  degraded.  (Gray  et  al.,  1981;  Havron  et  al.,  1979; 
Mirabella,  1978;  Shelnutt  et  al.,  1978)  found  that  such  is 
the  case  in  many  units.  Among  operational  units,  emphasis  is 
placed  on  passing  the  tests,  rather  than  discovering  and 
correcting  deficiencies  (Shelnutt  et  al.,  1978).  Leaders  are 
willing  to  cover  up  errors  and  to  do  the  tactically  safe 
thing  if  they  believe  their  performance  is  being  evaluated. 

As  described  by  Mirabella  (1978),  other  ARTEP 
applications  in  the  area  of  training  PMA  are  often  limited  by 
the  lack  of  quantitative  data.  He  states: 

One  of  the  philosophical  problems  with  the  ARTEP  is 
that  it  does  not  adequately  distinguish  between 
evaluation  for  training  diagnosis  and  evaluation  for 
accountability.  A  result  has  been  that  many 
commanders  regard  ARTEP  as  a  report  in  spite  of 
guidance  to  the  contrary  from  TRADOC.  (p.  2) 

Quantitative  data  can  provide  a  means  to  adequately 
diagnose  performance  (Shaket  et  al.,  1981).  Futhermore, 
quantitative  measures  can  provide  information  related  to 
training  objectives,  training  strategies,  and  training 
effectiveness . 

Hawley  and  Dawdy  (1981),  found  that  the  maintenance  of  a 
high  level  of  combat  readiness  requires  frequent  evaluations 
of  individual  and  unit  training  along  with  a  means  of  quickly 
diagnosing  and  remediating  performance  problems.  Moreover, 
timely  and  accurate  feedback  is  necessary  to  agencies 
external  to  the  units,  especially  those  involved  in  the 
design  and  development  of  the  publications  that  guide  the 
conduct  of  training  in  units.  A  number  of  research  efforts 
have  cited  problems  concerning  the  feedback  of  performance 
measurement  data  from  the  units  to  the  institutions. 

Burnside  (1981) ;  Ryan  and  Yates  (1977) ;  Scott  and  Ekstrom 
(1983);  and  Witmer  and  Burnside  (1982)  all  point  to  a  need 
for  better  feedback  loop.  Although  the  feedback  is  either 
subjective  or  a  mix  of  objective  and  subjective  data, 

Burnside  (1981)  found  that  subjective  feedback  (informal 
comments,  surveys/questionnaires,  and  interviews)  is  used 
most  often.  More  objective  feedback  (e.g.,  observation  of 
field  performance,  analysis  of  existing  performance,  and 
operational  field  performance  testing)  is  limited  due  to 
numerous  constraints  (e.g.,  availability  and  cost).  The 
general  findings  are  that  feedback  to  the  Army 
centers/schools  is  somewhat  disorganized  and  largely  based  on 
subjectively-derived  data;  there  is  a  lack  of  integrated  data 
collection  with  unit  activities;  and  that  ARTEP  results  are 
not  always  available  and  are  not  definitive  enough  to  provide 


9 


adequate  field  performance  feedback  (Burnside,  1981;  Scott  et 
al.,  1983). 

In  a  study  conducted  by  Witmer  and  Burnside  (1982), 
training  developers  indicated  that  the  feedback  they  receive 
from  the  field  via  training  evaluators  does  not  satisfy  all 
their  training  needs.  Feedback  was  found  to  be  lacking  in 
specificity  and  objectivity;  its  validity  was  perceived  to  be 
questionable.  Training  developers  also  pointed  to  a  lack  of 
interaction  among  the  divisions  of  the  DTD  and  between  DTD 
and  DOES.  These  findings  are  reiterated  in  the  Key  Decision 
Makers  (KDM)  survey  discussed  later  in  this  report. 

Skill  qualification  test.  The  SQT  is  designed  to  be  a 
validated  performance-oriented  test  of  soldier  ability  to 
perform  critical  tasks  required  by  his  Army  job  (Hill  et  al., 
1981) .  As  stated  by  the  US  General  Accounting  Office  (GAO) 
in  1982,  "it  is  the  Army's  only  diagnostic  tool  for  measuring 
individual  soldier  proficiency  in  critical  tasks."  The 
soldier  is  trained  to  specified  standards  (stated  in  the  SM) 
and  is  tested  against  those  standards  to  determine 
proficiency.  The  SQT  consists  of  three  components: 
on-the-job  testing,  hands-on  testing,  and  written  testing. 

In  1977  Spencer,  Klemp,  and  Cullen  reported  that  the 
reliability  and  validity  of  SQTs  was  "very  good"  since  it  was 
a  practical,  hands-on  test  of  specific  unit  performance. 

Five  years  later,  the  US  GAO  reported  that  the  SQTs  ability 
to  meet  the  Army's  needs  was  "questionable"  (US  GAO,  1982). 
This  was  confirmed  by  Simpson  et  al.  (1984).  To  a  large 
extent  this  is  due  to  the  fact  that  SQTs  are  becoming  less 
and  less  performance-oriented  (hands-on)  and  more  reliant  on 
written  pencil-and-paper  tests.  Although  O'Brien,  Harris, 
and  Osborn  (1979)  suggest  that  this  trend  tends  to  improve 
the  validity  and  reliability  of  the  tests  themselves,  other 
research  (Burnside,  1981;  Harman,  Steinheiser  and  Snyder, 
1978;  US  GAO,  1982)  suggest  that  there  may  be  a  concurrent 
decrease  in  the  relevance  to  actual  task  performance  in  the 
job  environment  by  the  soldier.  Furthermore,  Harman  has 
indicated  that  problems  have  been  most  frequently  found  with 
the  written  components.  In  fact,  soldiers  have  a  history  of 
a  high  level  of  success  on  the  two  other  components.  In 
spite  of  this,  SQT's  are  becoming  less  and  less 
performance-oriented. 

The  US  GAO  (1982)  reported  that  soldiers  were  not 
trained  in  all  critical  job  tasks.  Since  only  a  selected 
number  of  tasks  are  tested,  test  results  do  not  accurately 
indicate  the  soldier's  ability  to  perform  critical  job  tasks. 
The  study  also  reported  that  most  individual  training 
programs  within  Army  units  emphasize  only  about  30  critical 
individual  tasks  per  year  at  the  sacrifice  of  as  many  as  150 
other  tasks  which  are  just  as  critical. 
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There  is  a  recurring  perception  that  SQTs  are  a 
once-a-year-event  rather  than  the  culmination  of  training. 

At  best,  they  indicate  how  well  a  soldier,  at  a  given  time, 
can  perform  specific  soldier  manual  tasks.  Hill  and  Sticht 
(1981)  conducted  an  interview  of  commanders  at  the  battalion 
level  to  find  out  how  SQTs  were  utilized  at  this  level.  The 
commanders  pointed  out  that  a  cycle  of  SQT  training 
exclusively  designed  to  pass  the  test  was  common. 

Furthermore,  when  asked  how  often  in  the  last  year  they  had 
received  individual  training  in  SM  tasks, 

23%  (of  781)  El-E4s,  responded  that  they  only  received  this 
kind  of  training  in  the  period  between  SQT  notices  and  actual 
tests.  Few  soldiers  indicated  they  had  received  remedial 
training  after  the  test  (US  GAO,  1982) . 

Although  most  unit  training  is  test-directed  (i.e., 
occurs  in  preparation  for  some  evaluation  exercise) , 
subjective,  and  no-fail  testing,  there  is  a  less  obvious  form 
of  evaluation  that  is  used  by  those  managing  training  on  an 
almost  daily  basis.  It  entails  talking  to  people  and  finding 
out  how  they  are  getting  along,  problems  they  have 
encountered,  etc.  (Hill  and  Sticht,  1981) . 

Other  Performance  Measurement  Systems 

The  literature  also  points  to  other  training  PMA 
systems.  Although  they  are  not  directly  applicable  to  this 
effort,  three  training  PMA  systems  are  presented  in  this 
section  of  the  report  since  many  of  the  encountered  problems 
are  similar  to  those  found  with  SQTs  and  ARTEPs. 

Stoffer  (1981)  investigated  the  Navy's  Tactical  Aircrew 
training  and  found  limitations  with  its  air-to-air  combat 
mission  training  PMA  system.  Some  of  these  limitations  are: 
a  lack  of  specific  training  objectives,  a  lack  of  trend  data, 
certain  important  measures  were  unmeasured,  inadequate 
debrief  data  formatting,  and  a  lack  of  quality  control  over 
raw  performance  data. 


Allen  et  al.  (1982)  investigated  the  Platoon  Table  IX 
battle  run  which  is  administered  primarily  for  the  purpose  of 
documenting  platoon  competence  in  small  unit  tank  gunnery. 

The  battle  run  consists  of  a  set  of  tactical  scenarios  that 
portray  both  offensive  and  defensive  platoon  missions. 
Although  it  presents  representative  and  challenging 
situations  and  evaluation  is  based  on  a  variety  of  measures 
and  scoring  procedures,  the  battle  run  employs  only  two  (2) 
objective  measures.  It  lacks  procedures  for  collecting 
objective  performance  data  in  the  field  and  procedures  for 
processing  and  interpreting  obtained  performance.  Thus,  its 
validity  and  reliability  are  reported  as  being  questionable 
or  nonexistent. 


In  the  Air  Force ,  the  greatest  need  in  aircrew 
performance  assessment  appears  to  be  the  development  of  valid 
performance  criteria  (Vreuls  and  Woolridge,  1977) .  Although 
great  technological  advances  have  been  made,  there  is  a  lack 
of  objective  and  quantitative  standards  of  crew  performance. 
Many  variables  are  involved  and  often  they  are  too  difficult 
and  costly  to  measure.  Thus,  quantitative  performance 
criteria  have  been  defined  for  very  few  operational  tasks. 

As  clearly  stated  by  Pettit  and  Magruder  (1985),  "the 
goal  of  a  training  PMA  system  is  to  objectively  measure, 
evaluate,  provide  feedback,  and  manage  personnel  performance 
against  operational  requirements  "  (p.  408) .  The  studies  we 
have  previously  presented  support  the  contention  that  this  is 
not  the  case  within  the  military.  However,  the  literature 
postulates  solutions  to  the  training  PMA  problem.  These  are 
presented  in  the  following  section. 


RECOMMENDED  PERFORMANCE  MEASUREMENT  SYSTEMS 

Numerous  studies  (Allen  et  al.,  1982;  Bialek  and 
Brennan,  1979;  Biers  and  Sauer,  1983;  Breidenbach  and 
Brictson,  1981;  Cormier,  1984;  Fuller,  Waag,  and  Martin, 

1980;  Gibson,  1978;  Havron  et  al.,  1978;  Havron  and 
McFarling,  1979;  Havron,  Hawley,  and  Dawdy,  1981;  Kavanagh, 
Borman,  and  Hedge,  1986;  Mirabella,  1978;  Mode  and  Buletza, 
1985;  Nieva,  Fleishman,  and  Rieck,  1985;  Obermeyer  and 
Vreuls,  1974;  Richardson,  1983;  Simpson,  Gutman,  and  Jarosz, 
1984;  Shaket  et  al.,  1981;  US  General  Accounting  Office, 

1982;  Vreuls  and  Woolridge,  1977;  Waag  and  Knoop,  1977) 
provide  an  insight  into  how  to  improve  training  PMA.  The 
following  is  a  review  of  major  issues  brought  forth  by  such 
studies  and  of  present  systems,  particularly  in  the  training 
device/simulator  arena,  that  demonstrate  attempts  to  provide 
improved  measurement  systems. 

Ihe_grit?jjon-  Erofclgja 

The  training  process,  particularly  within  the  military, 
is  often  characterized  as  having,  early  in  the  process, 
extensive  measurement  and  relatively  well-developed  criteria. 
However,  as  one  gets  closer  to  the  operational  missions  of 
units,  there  is  less  measurement,  and  performance  criteria 
become  more  complex  and  harder  to  measure.  This  is 
particularly  applicable  to  performance  tests  where  one  or 
more  individuals  are  required  to  accomplish  a  job-related 
task  under  controlled  conditions.  The  more  the  problem  of 
defining  performance  criteria  is  unresolved  or  overlooked, 
the  less  valid  are  the  measures,  criteria,  and  decisions. 
(Vreuls  and  Woolridge,  1977) . 

The  literature  presents  some  attempts  to  resolve  this 
"criterion  problem."  Burroughs  (1985)  provides  criterion 
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performance  measures  for  reliable  tests  of  nonprocedural  Ml 
tank  driver  skills;  Steinheiser  and  Snyder  (1986)  pointed  out 
issues  related  to  criterion-performance  testing  which  should 
be  considered  when  developing  individual  and  weapon  crew 
tests;  Fuller  et  al.  (1980)  describe  the  Advanced  Simulator 
for  Pilot  Training  (APM)  system,  one  of  the  first  attempts  to 
develop  a  comprehensive,  real-time  measurement  capability  for 
a  research  simulator;  and  Mirabella  (1978)  proposes  the 
development  of  an  adequate  criterion-referenced  system  of 
evaluation  which  shifts  away  from  a  go/no  go  evaluation  to 
one  which  obtains  detailed  descriptions  of  behavior  involved 
in  two-sided  combat  simulation.  A  common  thread  seems  to 
guide  the  development  of  these  criterion-referenced  systems 
to  develop  performance  criteria  that  are:  more  objective, 
empirically  derived,  performance-oriented  rather  than 
process-oriented,  and  supported  by  an  analytical  model.  This 
is  further  supported  by  a  model  which  Medlin  (1979)  developed 
to  evaluate  unit  tactical  performance.  The  Combat  Operations 
Training  Effectiveness  Model  (COTEAM) ,  as  it  is  called,  was 
developed  using  the  ARTEP  evaluation  system  as  a  starting 
point.  It  modifies  the  current  ARTEP  manual  and  provides: 

1)  a  realistic  simulated  combat  environment  in  which  units 
perform  technical  operations  and  from  which  objective 
performance  data  can  be  obtained,  2)  procedures  for  defining 
standards  against  which  unit  performance  can  be  compared,  and 
3)  techniques  by  which  training  deficiencies  and  training 
level  combat  readiness  can  be  assessed. 

The  Air  Force  community  has  also  shown  concern  for  the 
need  to  develop  criterion-referenced  measures.  Hedge, 
Ballantine,  and  Gould  (cited  in  Blackburst  and  Baker,  1983) 
points  to  the  Air  Force  Human  Resources  Laboratory's 
(AFHRL's)  attempt  to  overcome  the  criterion  problem  by 
employing  a  variety  of  measuring  techniques:  (1)  a  task 
rating  form  where  specific  task  data  are  collected,  (2)  a 
dimensional  rating  form  where  task  dimensions  are  rated,  (3) 
a  global  rating  form  developed  to  collect  rater  overall 
impressions  of  first-termer  proficiency,  and  (4)  an  Air 
Force-wide  rating  form  developed  to  be  representative  of  all 
specialties  in  the  Air  Force.  Finally,  Kavanagh  et  al. 

(1986)  also  focus  on  criterion-development.  They  propose  a 
model  that  looks  at  input  variables  (e.g.,  individual 
characteristics),  process  variables  (e.g.,  cognitive  process) 
and  outcome  variables  (e.g.,  performance  measurement 
quality) . 


The  Job 


e  Approach 


It  is  not  practical,  cost  efficient,  nor  necessary  to 
measure  all  relevant  variables  for  all  tasks.  Therefore,  it 
is  important  to  be  able  to  sample  critical  tasks  and 
measures.  A  solution  to  this  issue  is  provided  by  the  job 
sample  approach  (Biers  and  Sauer,  1983;  Vreuls  and  Woolridge, 
1977) .  The  job  sample  approach  allows  for  a  reduction  of  all 
possible  measures  to  a  smaller,  representative  set  of 
measurement  candidates.  If  done  in  a  systematic  and 
empirical  manner,  it  can  be  a  valid  measure  of  per f ormance . 
For  example.  Biers  and  Sauer  (1983)  developed  job  sample 
tests  for  armor  crewmen  and  conducted  a  study  which  indicated 
that  the  linear  combinations  of  job  sample  test  measures 
accounted  for  a  very  high  proportion  of  variability  in  past 
armor  crewmen  success  at  Annual  Qualifications. 


Validity  and  Reliability 

Reliable  and  valid  measures  of  performance  are  necessary 
to  determine  if  trainees  have  acquired  the  skills  intended 
and  to  estimate  the  cost  effectiveness  of  the  training  system 
(Breidenbach,  Ciavarelli,  Sievers,  and  Lilienthal,  1986). 
However,  the  literature  review  shows  only  isolated  instances 
where  measurement's  validity  and  reliability  have  been 
determined. 


If  a  test  is  reliable,  it  is  consistent  and  stable 
between  measurements  in  a  series.  Inconsistencies  in 
measurement  can  often  be  attributed  to  variations  in  test 
content  or  test  situation,  subject  response  variation, 
variations  in  test  administration,  and  variations  in  the 
observation  process,  studies  have  been  conducted  which  point 
to  a  low  degree  of  agreement  between  raters  of  Army  job 
performance  tests  (Pickering  and  Anderson,  1976) .  some  of 
the  factors  that  are  considered  to  be  the  main  source  of  this 
disagreement  are:  some  performance  measures  appeared  to  be 
interpreted  differently  as  a  function  of  specific  unit 
Standard  Operating  Procedures  (SOPs) ;  the  evaluation  of 
several  performance  measures  was  dependent  on  the  examinee's 
verbal  report,  which  might  lead  to  a  situation  of  low 
reliability;  and  some  performance  measures  were  ambiguous 
statements  which  were  open  to  the  interpretation  and  bias  of 
the  individual  examiner. 


Validity  indicates  the  degree  to  which  the  test  is 
capable  of  achieving  certain  aims.  Tests  are  used  for 
different  purposes,  each  requiring  a  different  type  of 
investigation  to  establish  validity.  Content  validity  (does 
the  content  of  the  test  sample  the  kinds  of  things  about 
which  conclusions  are  to  be  drawn?) ;  criterion-related 
validity  (does  the  test  compare  well  with  external  variables 
considered  to  be  direct  measures  of  the  characteristic  or 
behavior  in  question?) ;  construct  validity  (to  what  extent  do 
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certain  explanatory  concepts  account  for  performance  on  the 
test?)  or  face  validity  (does  the  instrument,  or  the  face  of 
it,  appear  to  measure  what  it  claims  to  measure?)  can  be 
determined  via  measurement  validation.  Measurement 
validation  requires  substantial  empirical  data  collection  and 
analysis,  which  is  time  consuming  and  costly  (Vreuls  and 
Obermeyer,  1985)  and  as  indicated  by  Breidenbach,  et.  al. 
(1986),  "a  critical  review  of  the  literature  shows  that  there 
are  far  too  many  instances  in  which  training  PMA  systems  have 
been  hastily  developed  and  applied"  (p.  281) .  Thus 
measurement  validation  is  seldom  performed. 

Overall,  the  literature  concludes  that  validity  and 
reliability  can  be  raised  by  an  improved  test  development 
process  (Kavanagh  et  al.,  1986;  Mirabella,  1978;  US  GAO, 

1982;  Vreuls  and  Woolridge,  1977;),  a  planned  program  of 
measurement  testing  (Breidenbach  et.  al.,  1986),  and 
provision  of  additional  rater  training  (Fuller  et  al.,  1980; 
Havron  and  McFarling,  1979) .  Some  examples  of  systems 
developed  to  meet  the  need  for  a  reliable  and  valid 
measurement  system  are:  (1)  the  Advanced  Simulator  for  Pilot 
Training  (one  of  the  first  attempts  to  develop  a 
comprehensive,  real-time  measurement  capability  for  a 
research  simulator)  (Fuller  et  al.,  1980);  and  (2)  the 
Tactical  Aircrew  Training  System,  which  employs  an  air-to  air 
combat  mission  training  PMA  system  (Allen  et  al.,  1982). 


As  postulated  by  the  Army  Science  Board's  Summer  Study 
of  1985  and  Mirabella  (1978),  quantitative  data  can  provide 
accurate  assessment  of  training  effectiveness  or  efficiency. 
Quantitative  data  may  also  be  easier  to  communicate  and 
utilize  when  providing  feedback.  This  has  been  identified  as 
one  of  DoD's  needs  when  training  PMA  feedback  is  involved 
(Witmer  and  Burnside,  1982).  It  can  also  allow  for  improved 
data  collection  and  interpretation  of  performance  data  for 
qualification  and  diagnosis  (Allen  et  al.,  1982).  The 
authors  are  aware  of  efforts  to  improve  the  collection  of 
performance  data  for  later  interpretation  to  support 
diagnostic  appraisals  of  training.  The  development  of  the 
electronic  clipboard,  for  example,  will  enhance  the 
collection,  transmittal  and  use  of  more  quantitative 
performance  measurement  data. 

As  stated  by  Roscoe  (cited  in  Vreuls  and  Woolridge, 
1977),  training  PMA  does  not  have  to  be  automated  to  be 
objective,  reliable,  and  valid.  However,  the  advantages  of 
automated  data  collection  (Crawford  and  Brock,  1977;  Mode  and 
Buletza,  1985;  Simpson,  Gutman  and  Jarosz,  1984;  Vreuls  and 
Woolridge,  1977) ,  automated  performance  measures  (Breidenbach 
and  Brictson,  1981;  Hawley  and  Dawdy,  1981;  Obermeyer  and 


Vreuls,  1974) ,  and  automated  feedback  to  personnel  in  charge 
of  training  (Simpson  et  al.  1984;  Witmer  and  Burnside,  1982) 
have  been  widely  recognized.  Automation  can  also  result  in 
increased  precision  and  reliability  (Briggs,  1984) . 

Therefore,  many  systems  have  incorporated  some  type  of 
automation  in  their  training  PMA,  data  collection,  and 
feedback.  Artificial  Intelligence  (AI)  has  been  proposed  as 
a  means  to  assess  maintenance  tasks  (Richardson,  1983)  by 
incorporating  a  task  analytic  approach  to  develop,  specify, 
and  sample  specific  and  concrete  training  PMA.  Shaket  et  al. 
(1981)  have  proposed  a  rule-based,  event-driven  computer 
model  for  the  representation  of  small-unit  combat  engagements 
and  for  subsequent  evaluation. 

Additional  recommendations  are  provided  based  on  the 
problems  encountered  with  SQTs  and  ARTEPs.  The  GAO  (1985) 
postulated  among  other  things  the  need  to  use  SQT  assessments 
as  a  diagnostic  aid  to  improve  training  and  that  specific 
tests  should  not  be  announced  in  advance.  In  order  to 
overcome  major  ARTEP  weaknesses,  Shaket  et  al.  (1981) 
recommend  an  evaluation  system  based  on  the  ARTEP  system  that 
is  tutorial,  portable,  modular,  and  incrementally  modifiable 
and  integrated.  (One  effort  that  holds  the  promise  of 
improving  on  the  current  ARTEP,  and  overcoming  the  weaknesses 
cited  by  Shaket  et  al,  is  the  Computerized  ARTEP  Production 
System  (CAPS) .  The  conceptual  framework  of  CAPS  has  been 
formulated  and,  as  this  report  is  published,  the  Army  is 
preparing  to  implement  the  developmental  phase  of  CAPS.) 
Furthermore,  Havron  et  al.,  present  recommendations  to 
integrate  new  technologies  into  the  ARTEP  evaluation 
component  (e.g.,  engagement  simulation,  battalion  simulation, 
etc . ) . 


A  commonality  is  found  in  the  recommendations  and 
alternatives  presented  in  the  literature.  All  stress  how 
important  it  is  to  define  measurement  concepts,  data 
processing  concepts,  and  data  interpretation  concepts. 

Havron  and  McFarling  (1979)  summarize  this  by  indicating  a 
need  for  an  integrated  system  of  measures  and  criteria;  the 
allocation  of  functions  for  observation,  judgment,  and  data 
collection;  the  development  of  data  analysis  procedures;  and 
the  development  of  procedures  for  feedback  formulation  and 
utilization  of  results. 

SiffiVllfltlQn 

A  proposed  solution  to  the  problem  of  conducting  more 
frequent  readiness  evaluations  in  the  face  of  many  resource 
constraints  is  the  use  of  training  devices  instead  of  actual 
equipment  (Hopkins,  1975).  Although  Semple,  Cotton  and 
Sullivan  (cited  in  Vreuls  and  Obermayer,  1985)  indicate  that, 
"Most  existing  automated  human-system  training  PMA  subsystems 
are  so  poorly  designed  that  they  are  useless,"  simulators  can 
provide  benefits  to  training  PMA.  The  use  of  simulators  for 
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training  and  for  evaluating  can  result  in  increased  precision 
and  increased  reliability  (Briggs,  1984),  and  provide 
advantageous  training  features  (Crawford  and  Brock,  1977) . 
Some  of  these  features  are:  timely  correction  and  guidance 
of  learners  behaviors,  instructional  procedures  which  may  be 
modified  as  results  indicate  their  effectiveness,  early 
awareness  of  the  attainment  of  desired  achievement  levels, 
and  determination  of  skill  acquisition  rates  (Briggs,  1984). 
Unfortunately,  subjective  evaluation  techniques  and 
unstandardized  testing  situations  are  common  (Vreuls  and 
Obermayer,  1985) .  Briggs  examined  the  Navy  Stock  Lists  of 
Training  Devices  and  selected  training  devices  which  could 
provide  most  valuable  proficiency-related  information. 
Following  this,  interviews  were  held  with  training  personnel 
to  find  out  what  provisions  existed  for  measuring  performance 
in  the  simulators.  Overall,  it  was  discovered  that 
instructors  often  did  not  know  how  to  evaluate  performance, 
good  performance  tests  were  not  available,  and  equipment  used 
for  recording  and  evaluating  performance  was  usually 
inoperable. 


DISCUSSION 

Despite  ongoing  efforts  to  resolve  the  problems  that 
exist  within  the  Army  with  regard  to  the  measurement  and 
assessment  of  performance,  the  problems  noted  during  the  Army 
Science  Board's  1985  Summer  Study  prevail.  This  study  has 
captured  research  supporting  that  the  Army  has  achieved 
little,  if  any  success  at  developing  an  integrated  system  for 
measuring  training  performance  and  using  the  results  to 
improve  the  training  of  its  individual  and  units. 

Several  specific  problems  were  revealed  during  the 
effort.  There  is  still  a  great  deal  of  reliance  on 
subjective  measures  of  performance.  Evaluators  frequently 
measure  performance  against  standards  which  require  the 
evaluator  to  make  judgments  on  the  adequacy  of  performance. 
There  is  a  lack  of  quantitative  performance  management  data 
relating  to  training  objectives,  training  strategies,  and 
training  effectiveness.  Performance  measurement  is  process- 
oriented  rather  than  product-  or  performance-oriented. 

The  validity  and  reliability  of  the  training  PMA  system 
is  likely  questionable.  The  training  process,  particularly 
in  the  Army,  is  characterized  by  a  heavy  emphasis  on 
performance  measurement  and  relatively  well-developed 
criteria  early  in  its  development.  However,  as  one  gets 
closer  to  the  operational  environment,  there  is  less 
measurement,  and  performance  criteria  becomes  more  complex 
and  harder  to  measure.  Since  the  ultimate  goal  is  to  predict 
operational  performance,  the  more  this  problem  is  unresolved, 
the  less  valid  the  measures  are.  Moreover,  the  lack  of  an 
effective  feedback  loop  between  the  units  in  the  fields  and 
the  proponent  school  perpetuates  the  training  PMA  problem. 
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Future  research  efforts  should  be  pursued  to  resolve  the 
problems  mentioned  above.  As  presented  in  this  study,  many 
ongoing  problems  in  the  Army,  Navy,  and  Air  Force  attempt  to 
resolve  this  lingering  problem.  Future  research  directions 
should  investigate  means  to  more  empirical  data,  better 
analytic  methods,  and  measurement  standardization;  assess  the 
accuracy  of  subjective  feedback;  and  develop  methods  for 
managing  and  utilizing  feedback  information.  Emphasis  should 
be  placed  on  the  application  of  learning  principles  such  as 
knowledge  of  results  and  retention  of  learning 
(retent ion/leaming  rates)  to  training  PMA.  Consideration  of 
such  basic  principles  can  result  in  both  increased  precision 
and  increased  reliability. 
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Department  of  Evaluation  and  Standardization 
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Directorate  of  Training  Development 

Defense  Technical  Information  Center 

Educational  Resources  Information  Center 

General  Accounting  Office 

Key  Decision  Maker 

Military  Occupational  Specialty 

Performance  Measurement  and  Assessment 

Project  Manager  for  Training  Devices 

Soldier's  Manual 

Soldier  Manual -Common  Tasks 

Skill  Qualification  Test 

Table  of  Organization  and  Equipment 

US  Army  Training  and  Doctrine  Command 

Training  and  Performance  Data  Center 

TRADOC  Systems  Analysis  Agency 

Training  Effectiveness  Analysis 
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GLOSSARY 


TERM 

Collective  Training 
TRADOC  Cir  350-3 


Criterion  Referenced  Test 
TRADOC  Cir  350-3 


Individual  Training 
TRADOC  Cir  350-3 


Institutional  Training 
TRADOC  Cir  350-3 


Objectivity 
TRADOC  cir  350-3 


Performance  Assessment 
MIL-HDBK-220B 


Training,  either  in  institutions 
or  units  that  prepares  a  group  of 
individuals  (crews,  teams,  squads, 
platoons,  etc.)  to  accomplish 
tasks  required  of  the  group  as  an 
entity. 

A  test  which  measures  what  an 
individual  must  be  able  to  do  or 
must  know,  in  order  to 
successfully  perform  a  task.  An 
individuals'  test  performance  is 
compared  to  an  external  criterion/ 
prespecified  performance  standard 
which  is  derived  from  an  analysis 
of  what  is  required  to  do  a 
particular  task. 

Training  which  the  individual 
officer,  NCO,  or  enlisted  person 
receives  in  institutions,  units  or 
by  extension  self-study,  that 
prepares  the  individual  to  perform 
specified  duties  and  tasks  related 
to  the  assigned  MOS  and  duty 
position. 

Training,  either  individual  or 
collective,  conducted  in  schools 
(Army,  service  school,  USAR 
school,  NCO  Academy,  unit  school) 
or  Army  Training  Centers. 

In  testing,  the  degree  to  which  a 
test  is  scored  the  same  by  two  or 
more  scorers  acting  independently. 

The  instructor  synthesizes  all 
performance  measurement 
information  to  assess  trainee 
performance.  The  performance 
measures  may  be  objective  (e.g., 
machine  generated  information  such 
as  number  of  target  hits)  or 
subjective  (e.g.,  information 
gathered  through  the  instructor 
senses  as  proper  communication 
format  used) . 


25 


Performance  Measurement 
MIL-HDBK-220B 


Process  Task 
TRADOC  Cir  350-3 


Product  Task 
TRADOC  Cir  350-3 


Qualitative 


Quantitative 


Reliability 
TRADOC  Cir  350-3 


The  scoring  of  trainee  proficiency 
either  subjectively  (e.g., 
instructor  opinion)  or  objectively 
(e.g. ,  automatic  computer 
measurement) . 

A  task  which  consists  of  a  series 
of  steps  resulting  in  the  soldier 
obtaining  a  single  discrete 
result.  The  task  is  evaluated  by 
observing  the  process  and  by 
scoring  each  step  or  element  as  it 
is  performed  in  terms  of  sequence, 
completeness,  accuracy,  or  speed. 
Examples  are  "put  on  the 
protective  mask"  and  "take  oral 
temperature. " 

A  task  which  terminates  in  a 
discrete  product  or  outcome  which 
is  observable  and  measurable.  The 
task  is  evaluated  by  looking  at 
the  product  or  outcome  in  terms  of 
completeness,  accuracy,  tolerance, 
clarity,  error  or  quantity. 

"Repair  the  carburetor"  could  also 
be  an  example  of  a  product  task. 

A  term  describing  a  performance 
measurement  standard  that  relies 
on  objective  ratings  or  word 
descriptions  to  determine  adequacy 
of  performance. 

A  term  describing  a  performance 
measurement  standard  that  relies 
on  numbers  to  determine  adequacy 
of  performance. 

The  degree  to  which  a  test 
instrument  can  be  relied  upon  to 
yield  the  same  result  upon 
repeated  administrations  to  the 
same  population. 
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Soldier's  Manual 
TRADOC  cir  350-3 


Unit  Training 
BTMSRC  83-1 
Validity 


A  manual  that  lists  for  the 
soldier  those  critical  tasks 
needed  to  perform  satisfactorily 
at  his  present  skill  level.  In 
addition,  the  SM  tells  the  soldier 
how  to  perform  the  tasks,  the 
expected  conditions  under  which 
they  will  be  performed,  and  the 
standards  which  must  be  met.  The 
SM  is  the  basis  for  the  tasks  used 
in  the  SQT. 

Training,  individual  or 
collective,  conducted  in  a  unit. 
The  degree  to  which  a  test 
measures  what  it  purports  to 
measure  (Handbook  in  Research  and 
Evaluation) . 
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EESKL 

Behavioral  Sciences  Research  laboratory 

CIV 

Cavalry  Fighting  Vehicle 

C&GSC 

Command  and  General  Staff  College 

CTEA 

Cost  and  Training  Effectiveness  Analysis 

GAO 

General  Aoocunting  Office 

HumRRO 

Human  Resource  Research  Organization 

I/ITEC 

Interservice/Industry  Training  and 
Equipment  Conference 

ISO 

Instructional  Systems  Development 

¥DR 

Knowledge  of  Results 

miles 

Multiple  Integrated  laser  Engagement 
System 

m  IRATE 

Project  Manager  for  Training  Devices 

ROI 

Return  on  Investment 

SL 

Skill  Level 

TEC 

Training  Extension  course 

TRADOC 

Training  and  Doctrine  Command 

USAREUR 

United  States  Amy  Europe 

USMC 

United  States  Marine  Corps 

A-5 


•VN0P9CS 

Seq:  01 

10  tkaba*:  NTEC  72-C-0033-1;  8D-A022  029 

Doit:  Nov  1073 

Title:  Advanced  Officer  Tactics  Training  Device  Needs  and 

Performance  Hoosirsment  Technique 

Author:  Hammell,  T.,  Oasteyer,  C.,  t  Pesch,  R. 

Corporate  Author:  General  D/iamics 

Summary:  Attempts  to  determine  advanced  tactics  training  device  needs 

for  sttearine  officers  <rtd  to  develop  a  technique  for  the 
measurement  of  tactical  training  performance. 

Key  Points:  Recoosends  computer-assisted  performance  measurement 

techniques  based  on  mathematical  eeapons  system 

effectiveness  (USE)  model. 


8eq:  02 

ID  Nuaber:  flFHRL-TR-70-57;  RD-R0G6  888 

Date:  Rug  1080 

Title:  Advanced  Simulator  for  Pilot  Training:  Design  of  Automated 

Performance  Heasuremont  System 

Author:  Fuller,  J.,  Uaag,  U.,  I  Hart  in,  E. 

Corporate  Ruthor:  Operations  Training  Division  -  AFHFL 

Summery:  Describes  current  status  of  the  automated  performance 

measurement  system  (flPtl)  In  the  Advanced  Simulator  far  Pi  lot 
Training  (ftSPT).  APR  eas  developed  to  meet  the  need  for  an 
objective  pilot  measurement  system. 

Key  Points:  APN  system  represents  one  of  the  first  attempts  to  develop  a 

comprehensive,  real-time  measurement  capability  for  a 
research  simulator.  Tdtes  criterion-referenced  approach  to 
measurement  definition. 


Seq:  03 

ID  Haber:  Not  Available 

Date:  Oct  1977 

Title:  Aircree  Performance  Heasuremont  (paper  presented  at 

Productivity  Enhancement:  Personnel  Performance  Assessment 
In  Navy  Systems  Syiposlia,  San  Diego,  CR) 

Ruthor:  Ureuls,  D.,  %  Uoolridgo,  L. 

Corporate  Author:  Canyon  Research  Qrocp,  Inc. 

Siamary:  Locks  at  Aircree  performance  measurement  in  terms  of: 

aircree  environment,  an  approach  to  measurement  development, 
and  future  research  needs. 

Detailed  account  of  aircree  performance  measiremen 


Key  Points: 


t. 


HAMDOM 

fTnUrlM 

Seq:  04 

(0  Huber:  HRI-flM  -57-6;  AD-W51  224 

Da it:  Apr  1957 

Tltlt:  Analysis  of  Efficiency  Ratings  Bastd  on  30-59  Days  of 

Observation  in  a  Coebat  Zone  and  in  the  Zone  of  interior 

Author:  Mot  tollable 

Corporate  Author:  AO  Office 

Sueecry:  Coaparas  short- ter*  efficiency  reports  eith  long  ter* 

efficiency  reports  to  detereine  if  an  increase  in  the  number 
of  evaluations  increases  the  nueber  of  officers  eith  overall 
Efficiency  Indexes. 

Key  Points:  necoeesndi  sore  frequent  evaluation. 


8eq:  05 

ID  Nueber:  PAI-BH-95-3;  RD-A150  149 

Oata:  Jan  1905 

Title:  faalysis  of  TOU  Gunnery  Training 

Author:  taxay,  J.  L. 

Corporate  Author:  Mel  Ionics  Systeas  Development 

Sueeary:  Presents  description  of  current  TOU  gunnery  training  and 

discusses  the  iepi  (cations  of  cr  analysis  of  this  training 
for  Identifying  improvements  and  alternatives.  Aeport 
presents  an  atteapt  to  evaluate  training  eater ials  (lesson 
plans,  POI's). 

Key  Points:  Aeoo**endt  analysis  of  training  eater  ials  (POI’s  and  lesson 

plans)  and  Interviews  and  observation  of  training. 


8eq: 

06 

10  Nueber: 

AAI-AH-77-21;  AO-A077  939 

Oata: 

Sep  1977 

Title: 

Analytic  Approach  to  Estimating  the  General  liability  of  Tank 

Cr*e  Performance 

Author: 

Boycan,  0.  0.,  t>  Rose,  A.  N. 

Corporate  Author: 

ARI 

fe-ary: 

Attempts  to  ieprove  general izabi 1 Ity  of  gunnery  tables 
(training  or  testing  exercises  which  define  Army  programs 
for  gunnery  training  and  tank  era*  evaluation). 

Key  Points: 

Performs  cluster  analysis  of  performance  objectives  to 
establish  commonality  and  general  izabi  1  ity. 

SVN0P9ES 

Saq:  07 

10  N iabr:  flRl-flN-77-10;  AD-A077  037 

Data:  Nov  1077 

Titla:  Analytic  Training  Effectiveness  Analysis  for  a  CTEfl  Updata 

Author:  Finlay,  0.  L.,  4  Treat*  I  a,  T.  A. 

Corpora ta  Author:  flfil 

Scasary:  Looks  at  too  alternative  training  coneapts  to  developing 

organizational  and  oparator  naintonanca  training  devices. 

Kay  Points:  Derives  relative  training  effectiveness  valuas.  Aalativa 

North  <AU)  ■  Relative  Effectiveness  (R£>/Aelative  Cost  (AC>; 
AE  ■  Effact  of  Altamotive/Effact  of  Bosa  Casa;  AC  ■  Cost  of 
Altamati  ve/Cost  of  Bosa  Casa. 


•  06 

ID  Nuabar:  flRI-RH-81-17;  RD-A128  070 

Data:  Nay  1061 

Titla:  toolytical  Nodal  for  Developing  Objaetiua  Haasuras  of 

AlrCraa  Proflclancy  alth  Nultl  war  iota  Tlaa  Baquancad  Data. 
Uoluna  I 

Author:  Connelly,  E.,  Johnson,  P.,  t  Shiplay,  B.  JT. 

Ccrporata  Author:  Parforoanca  Noasuraaant  Assoc  Iotas 

Suaaory:  Prasants  thaoraticol  investigation  of  analytical  aathods  for 

deriving  diffarantial  easting  functions  froa  pra-salactad 
soap I as  to  eultivorlate,  tlaa  saquancad  observations  of 
alrcras  parforaanca.  Nodal  output  Is  a  sat  of  sailings. 

Kay  Points:  To  isprova  NOE  aircres  training  prograas,  naad  battar 

quality  data  in  tha  evaluation  of  aircraa  proficiancy. 
Racoaaends  usa  of  fully  instruaantad  training  aith  graatar 
I  avals  of  datail  in  parforaanca  aaasuroaant.  Purports  usa 
of  autoaatad  aodal . 


Saq:  00 

10  Ncabar:  HMM36  000 

Data:  Jan  1064 

Titla:  Analytical  Nodal  of  Looming  and  Parforaanca  of  Araor 

Author:  8tleha,  P.  6  Edaards,  T. 

Corporata  Author:  Dacisions  and  Desipw,  Inc. 

Suaaory:  Docuaants  devalopaant  and  faatu'es  of  a  aodal  to 

inuastigata  issuas  regarding  tha  acquisition  and  ratantion 
of  coaplax  aiiitary  skills. 

Kay  Points:  Dacisions  regarding  tha  acnogaaant  of  training  are  aost 

affactiua  ahan  supported  by  Inforaation  about  the 
effactivanass  of  diffarant  training  options.  Hosavar,  this 
process  is  ganarally  expansive  and  full  of  serious 
aethodo logical  and  practical  probieas. 


tuynMCfi 

lynurvu 

toq: 


ID  Niaber: 

Dai*: 

Till*: 


Author: 

Corporal*  Author: 


Kay  Points: 


10 

flRI-flh-04-32;  AD-A138  222 
Fob  1004 

Antecedents  and  Consaquaneas  of  Parforewtca  Feedback  In  an 
Organizational  tatting 

llgen,  0.,  Fisher,  C.,  Ougonl,  B.,  Hot tea,  H.  %  Taylor,  t. 
Pirdua  University 

Major  focus  on  critical  rola  played  by  faadbock  in  oorfc 
anvironeants.  Rasacrch  consisting  of  1 1  taroUr*  rout  as  on 
parforaanc*  faadbock,  developeant  of  Instnaent  to  aaasura 
faadbock,  and  rasaarch  of  supervisors'  sillingnass  to 
provide  faadbock. 

Eaphasizas  rola  of  faecfeock  in  tha  aotivationol  process, 
individual  dlffarancas,  and  diffarant  rotas  that  supervisors 
play. 


8oa: 

ID  Muabar : 

Data: 

Titla: 

Author: 

Corporate  Author: 
SuMary: 


Kay  Points: 


11 

flRI-flM-81-13;  AD-A127  090 
Jul  1061 

Implication  of  Auie-based  Coaputer  Model  to  tha  Evaluation 
of  Coabat  Training:  A  Feasibility  Study 
Shaket,  E.,  Sal ah,  J.  0  Fraady,  A. 

Parcaptronics 

Exaaines  the  feasibility  of  a  rule-based,  event  (ft*  I  van, 
coaputer  aodal  for  tha  representation  of  seal  I  unit  coabat 
engagaeants  and  for  subsequent  parforaanc*  evaluation. 

Major  flRTEP  eeakness:  no  objective  say  to  deteralns  tor* Inal 
aission  outcoaes.  Several  trng  systees  overcoae  this 
deficiency  (SCOPES,  REALTRRIH  and  MILES).  Evaluation  systea 
should  be  tutorial,  portable,  aodular,  I ncr seen  tolly 
eodifiabia,  ft  integrated. 


Seq: 

ID  Muabar: 

Dote: 

Title: 

Author: 

Corporate  Author: 
Suaaary: 


Kay  Points: 


12 

ARI-TP-36I;  RD-A075  410 
Jul  1070 

^plication  of  Tactical  Engage* wit  Slaulation  for  Unit 

Proficiency  Mecmraaent 

Krtarr,  C.  H.,  Root,  R.  T.,  ft  Uord,  L.  E. 

RRI 

Revises  application  of  a  tactical  training  systea  called 
Tactical  Engogeeant  Slaulation  <ES>.  ES  uses  objective 
accurate  assessaent  techniques  and  provides  roalistic 
tactical  training. 

Points  out  need  to  aeasire  teaa  and  unit  proficiency,  tha 
lack  of  knoeledge  in  this  area,  and  existing  Arey  coabat 
unit  parforaanc*  aeasureaent  techniques  shich  depend  largely 
on  Judge into I  data.  Presents  relevant  literature. 


WORSES 


llOftefcer. 

Oat*: 

Tit!*: 

Author: 

Corporate  Author: 
SUMary: 


Kay  Points: 


13 

87?  17-18E-JB 
hoy  1884 

Araor  Cro— an  Job  Book  H06  18E10/20 
Hot  Appl icobl* 

US  Arey  Araor  School 

Used  os  an  HCO  training  aanogaMnt  tool  to  raoord 
da*onstrot*d  prcflciancy  on  soldier's  aanuat  (all  coaeon  and 
NDS-specIfic)  tasks  for  oh  I  eh  th*  SL  1  or  2  soldier  is 
responsible.  Provides  space  for  st^arvisor  to  record  90  or 
no-go  and  date. 

Basic  idea  is  good,  but  doifctful  that  it  is  utilized  eith 
any  degree  of  effectiveness. 


8eq: 

ID  fleeter: 

Dote: 

Title: 

Author: 

Corporate  Author: 
Sueeary: 


Key  Points: 


14 

8TP  17-lflK-JB 
Sep  1884 

fb-eor  Creeean  Job  Book,  hOS  18k 10/20 
Not  Applicable 
US  Aray  Areor  8chool 

Used  as  an  NCO  braining  eanogeeent  tool  to  raoord 
deeonstraied  proficiency  on  soldier's  eanuol  (all  coeeon  and 
flOS-specif  ic>  tasks  for  shich  the  9L  I  or  2  soldier  Is 
responsible.  Provides  space  for  supervisor  to  record  go  or 
no  and  date.  - 

Basic  Idea  Is  good,  but  doubtful  that  It  is  utilized  eith 
any  degree  of  effectiveness. 


Seq. 

ID  Nueber: 

Oats: 

Title: 

Author: 

Corporate  Author: 
Sueeary: 


Key  Points: 


15 

AR I -AN-64-84;  AO-A142  534 
JUn  1884 

Areor  Training  in  Coabat  Units.  Final  Asport  Uoluee  II: 
Training  Products 

Siapson,  H.,  NoCollva,  H.,  8  Fuller,  R. 

Anoccpo  Sciences,  Inc 

hethods  developed  to  select  and  prioritize  areor  eras  tasks, 
and  to  define  the  scope,  content,  and  eathods  to  eaploy  in 
areor  eras  individual  training. 

Points  out  Arey  unit  problems  and  recoeeendf  solutions. 


A-10 


SVKP8ES 

U*: 

10  Number: 


Oat a: 

Titla: 

Author: 

Corporata  Author: 


Kay  Points: 


16 

AAIHR-1216;  AD-A075  394 
JUt  1079 

frmor/Antl-frmor  Tool  Tactical  Performance 
Scott,  T.  0.,  Hal  In,  L.  L.,  S  Hardy,  0.  0. 

mi 

Successful  wilts  (re inf  tank  pits)  as  opposed  to 
unsuccessful  units,  ora  characterized  by  sound  tactical 
parforeanee  In  planning  the  attock,  initial  deplo^ent,  use 
of  cooar  and  conceal  sent,  surveillance  and  use  of  flrapoeer . 
Efficiency  (econoeical  training)  eay  result  by  concentrating 
on  eissions  requiring  satisfactory  execution  of  eldest 
oariety  of  coeeon  skills. 


Seq: 

10  HMm'  : 

Date: 

Title: 

Author: 

Corporate  Author: 
Summary: 


Key  Points: 


17 

mi-TR-642;  PD-A199  494 
Jul  1084 

Arey  Maintenance  Training  and  Evaluation  Simulation  Systee 
(flHTESS)  Device  Evaluation:  Uol  I.  Overview  of  Study  Effort 
Uhger,  K.,  Seezey,  A.,  Hays,  R.  S  Mirabel  la,  A. 

SAI  _ 

Overviee  of  WfTESS  program;  This  report  is  uol  1  of  3  volume 
series.  Summarizes  quantitative  (transfer  of  training)  and 
qualitative  (user  opinion)  data  about  2  prototype  devices. 
Objective  of  AHTE88:  provide  cost  and  training  effective 
maintenance  simulators  that  can  be  easily  adapted  to  a 
variety  of  maintenance  tasks  across  a  number  of  HOSs. 


Seq: 

10  Number: 

Date: 

Title: 


Author: 

Corporate  Author: 
Summary: 


Key  Points: 


18 

mi-PA-643;  AD-A198  916 
Jul  1884 

Army  Maintenance  Training  and  Evaluation  Simulation  System 
(flHTESS)  Device  Evaluation:  Uol  II.  Transfer  of  Training 
Assess. of  2  Prototype  Devices 
Uhger,  K.,  Seezey,  A.,  Hays,  A.  t  Mirabel  la,  A. 

SAI 

Transfer  of  training  study:  students  from  several  HOSs  mere 
trained  on  conventional  method  or  1  of  2  AHTE38  devices  end 
subsequently  tested  on  operational  equipment. 

Comparison  be  tee  an  conventionally  trained  students  and 
device  trained  students  Indicated  statistically  superior 
performance  by  90S  of  conventionally  trained  students. 
Transfer  of  training  Index  (E/C  ratio)  Indicated  high  level 
of  transfer  in  all  cases. 

mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmrnmmmmmmm 


A-ll 


«M»n 

Uq: 

ID  H later: 

Data: 

Till*: 


Author: 

Corporate  Author: 


Kay  Points: 


19 

AAI-TA-589;  AD-A146  237 
Oct  1083 

fray  Aointananoa  Training  and  Evaluation  8y*tae  (AMTESS) 

Davies  Oavalopaant  and  Features 

Crisaoll,  E.,  Uhga r,  K.,  Soazey,  A.  I  Hoyt,  A. 

8AI 

Ovarviee  ofthe  develops ant  and  tasting  of  fVTTESS  devices. 
Basad  on  AMTESS  dociaontot  ion  and  data  gathered  in 
structured  intarviees  of  AMTESS  pro j act  participents’ 
opinions  on  spaeifie  features. 

One  of  Investigated  faaturas  ■  parforaanca  fa attack.  host 
Intarvlaaaas  fait  faadback  giuan  to  studant  on  studant  CRT 
sos  good.  Qraatast  problaa  sith  both  devices  (Gruaaan  and 
Soul  I  la/Burtak)  oas  dapondcfci  t  Ity.  Doon-tiae  ploguad 
device  evaluation. 


8aq: 

ID  Nuaber: 

Oats: 

Titla: 


Author: 

Corporate  Author: 


Kay  Points: 


20 

ARI-TR-644;  AD-A159  517 
Jul  1084 

fray  Maintenance  Training  and  Evaluation  Systan  (AMTESS) 
Davica  E^iluotlon 

Unger,  K.,  Seezay,  R.,  Hays,  A,  I  Mirabel  la,  A. 

8AI 

8tudants  froa  savarol  HOSs  sara  trained  to  perfora 
aointananoa  tasks  by  using  one  of  2  prototype  tng  dav. 
Opinion  data  oas  col  I  acted  on  the  odaquooy  and  effectiveness 
of  the  devices. 

Intarvlaaaas  pointed  out  features  they  liked:  request  help 
f so  tire,  faadback,  absence  of  instructor,  and  procedural  izad 
self-paced  lessons.  Both  devices  ears  coaposed  of  4  aojor 
coaponants:  studant  station,  instructor  station,  and  2  X 
eodules. 


8eq: 

ID  tfeflbar: 

Date: 

Title: 

Author: 

Corporate  Author: 
8usaary: 


Kay  Points: 


21 

PB-82- 186772 
March  1982 

Arey  Heeds  to  Modify  its  Systae  for  Measuring  Individual 
8oldior  Parforaances 
Oonaral  Accounting  Office 
U.8.  Oonaral  Accounting  Office 

Report  suoaarizes  Oonaral  Accounting  Office's  (0A0>  concern 
on  the  extant  to  ehlch  the  fray's  SQTs  eeasire  soldier 
proficiency  and  identify  individual  training  needs. 

Cost  Effectiveness  of  SOT  developeant  and  administration  Is 
questionable.  8QT  prograa  (in  its  design  and 
ispleaentatlon)  does  not  east  fray’s  need  to  eaosure 
individual  proficiency  in  order  to  identify  tmg  needs. 


A-12 


SVnUnjtS 


tea* 

ID  Number: 

Oats: 

Title: 

Author: 

Corporate  Author: 
Summary: 


Keg  Points: 


22 

AD-A118  990 
Sep  1982 

Army's  Initiative  to  leprous  the  Skill  Qua I  if (cation  Test 
Prop-am  flag  Not  Achieve  Proeised  Changes 
General  Accounting  Office 
U.S.  GAO 

Follor-on  to  “Army  Needs  to  Nodifg  Its  Sgstees  for  Neasuring 
Individual  Soldier  Proficiency*.  Coeplaint:  nothing  has 
been  done  to  fotloe  recoeeendations  of  original  report. 

North  finding  out  hoe  euch  has  be*t  done  or  changed  since 
the  original  report  eas  or  it ten. 


Saq: 

ID  Number: 

Date: 

Title: 


Author: 

Corporate  Author: 
Suemary: 


23 

RFVHL -TP-83-28;  RD-A133  982 
Sept  1983 

Artificial  Intelligence:  Art  Analysis  of  Potential 
Applications  to  Training,  Performance  Heasureeent  aid  Job 
Per foraance  Aiding 
Richardson,  J.  J. 

Denver  Research  Institute 

Points  out  to  hoe  Al  could  be  used  in  the  area  of 
PerVormance  Heasureeent. 


Keg  Points:  Presents  definition  of  perf .  eeas.  roles.  Use  of  Job  task 

proficiency  tests  ccn  cut  doen  on _  ealntenance  training 
costs.  Nature  of  tasks  could  facilitate  use  of  Al  in 


Seq: 

10  Nueber: 

Date: 

Title: 

Author: 

Corporate  Author: 
Summary: 


Keg  Points: 


24 

ARI-RR-1199;  RQ-A060  983 
Aug  1978 

Assesseent  of  the  Navigation  Performance  of  fray  Aviators 

Under  Hap-of-The  Earth  Conditions 

Fineberg,  N.  L.,  tie  Is  ter,  0.,  8  Farrell,  J.  P. 

ARI 

Obtained  eaplrical  data  on  hoe  the  NOE  navigational  skill 
level  of  Army  evaluators  Is  affected  by  pilot  experience  and 
different  levels  of  training.  Defines  baseline  on  pilot 
navigation  proficiency  and  a  methodology  to  measure  NOE 
flitfit  performance. 

Concludes  that  all  pilots  need  more  NOE  training. 

Recommended  training  should  emphasize  practical  exercises 
rather  than  standard  lecture  techniques. 


AUNnpxfi 

wnunso 

28 

Not  Aval lablo 
Not  Available 

Basie  FII0>t  Training:  Introductory  Training  by  Naans  of  a 
Simulator 
Fehler,  F. 

Psychology  Uhlt,  Osraan  Pray  School 

Study  conducted  to  set  u?  a  scraaning  prograa  to  distinguish 
bataoan  fast  and  slot  laamsrs.  Concantrotas  on  tha  job 
soap  I  a  approach  (J6A).  JSA  used  with  UH-1D  si  out  a  tor  at 
Aray  Aviation  School. 

Points  out  that  current  aviator  training  oathodologias  ara 
vary  traditional  olthou^i  tasks  <rs  bacoaing  aora  coaplax 
dua  to  hi£t  tach  dav.  Hathodologias  concentrate  on  basic 
aanauvars  ahich  ara  not  vary  pradictiva  of  final  out  coats. 


26 

AR I -TP-353;  AD-A070  099 
Her  1979 

Battalion  Coaaand  Group  Performance  in  Siaulatad  Coabat 
Kaplan,  I.,  t  Barber,  H. 

mi  _ 

Describes  tha  application  of  the  ARTEP  coaaand  group  module 
to  measurement  of  coemand  group  performance  in  the  Combined 
Arms  Tactical  Training  Simulator  (CATTS). 

Heasurement  techniques  developed  in  this  project  used  to 
provide  feedback  to  coemand  groups  trained  in  CRTTS  aid  to 
Investigate  training  effectiveness  of  battle  siaulations. 
Paragraph  and  figures  on  ARTEP  -  shot  it's  ail  about. 


Seq: 

27 

ID  Number: 

Not  Available 

Oate: 

1995 

Title: 

Behavioral  Analysis  and  Heasurement  Nethods 

Author: 

Heister,  D. 

Corporate  Author: 

Not  Available 

Summary: 

Good  compendium  of  methods  used  to  study  eork  performance, 
ranging  from  analytic  methods  used  in  system  development  to 
observation,  ratings,  surveys,  field  measurement,  self 
reports,  and  interviews. 

KOy  Points: 

Good  source  of  Info  an  Oelphl,  Policy  Capturing,  and 
Performance  Neasur meant. 

6eq: 

ID  NurtW': 

Date: 

Title: 

Author: 

Corporate  Author: 
SuoMry: 


Key  Points: 


Seq: 

10  Number: 

Date: 

Title: 

Author: 

Corporate  Author: 
flummery: 


Key  Points: 


A-14 


8VN0P9ES 

8eq: 

10  Amber: 

Date: 

Title: 


Author: 

Corporate  Author: 


Key  Points: 


28 

9E9AL-A8-69-7;  AD-R077  74 1 
Hay  1000 

BE8RL‘s  Field  Laboratory  Studies  in  Hunan  Porforeance 
Exper  i  eentot  i  on . 

H\pMn,  A.,  Sternberg,  J.  J.,  S  Banks,  J.  H. 

BE3AL 

Summarizes  proyess  in  huean  porforeance  experimentation  in 
too  on-going  eork  units  sithin  the  Coobat  Research  Division: 
Huean  Performance  Experimentation  in  night  operations  and 
Dependable  Porforeance  in  ton  I  tor  Jobs. 

Rtteepts  to  discover  general  principles  ehich,  then  applied 
operationally,  el II  enhance  the  porforeance  of  individuals 
•I thin  the  systee. 


8eq: 

10  Nu*w: 

Date: 

Title: 

Author: 

Corporate  Author: 
Sinewy: 


Key  Points: 


29 

tot  Available 
1974 

Cap  tiring  Judgeent  Policies:  R  Field  Study  of  Porforeance 
f^praisal 

Taylor,  A.,  SUllsted,  U. 
tot  Available 

Coepares  eatheeatical  eodels  of  judgeent  policy  by 
evaluating  629  porforeance  reports  during  a  single  rating 
scale. 

Field  study  of  porforeance  appraisal  policy  of  cadet  rating 
at  U.S.  Force  Academy.  Discusses  policy  cap  tiring 
applications. 


8eq: 

10  Nuebor: 


Oats: 

Title: 

Author: 

Corporate  Author: 


KOy  Points: 


30 

AFWL-TR-74- 106  <U1> 

1977 

Capturing  Rater  Policies  for  Processing  Evaluation  Data 
Zedeck,  8.  8  Kafey,  D. 

Organizational  Behavior  6  Huean  Perform. 

Investigates  rater's  strategy  or  policy  for  assessing 
information  on  nine  criterion  elements  for  a  ratee. 
Utilizes  Judgment  Analysis  (JAN)  procedure. 

Utilized  JAN  technique.  Hypothetical  behavioral 
descriptions  of  nurses  sere  presented  to  67  nursing 
personnel  (raters). 


A-15 


tuynaAM 

VtiW 

teq: 

ID  Nu*er: 

Date: 

Title: 

Author: 

Corporate  Author: 
Summary: 


Key  Points: 


31 

AD-3005  321 
Dec  1374 

Combat  Aeodg  Cree  Perforeance  Heasirement  System:  Phase 
MIC  Design  Studies. 

Obermayer,  A.,  I  Urmuls,  D. 

Homed  Systems  Sciences,  Inc. 

Atteepts  to  I  sprout  training  performance  information  by 
providing  a  definition  of  information  and  developing 
methods  for  meonrement.  Report  deals  mlth  mays  to 
determine  features  of  system  that  meets  research  needs 
previously  reported. 

Hoe  to  ieprove  training  performance  information.  Based  on 
extensive  research. 


8eq: 

ID  Niabm: 

Date: 

Title: 

Author: 

Corporate  Author: 
St  merry: 

Key  Points: 

32 

A0-A101  303 

JUI  1370 

Combat  Effective  Training  Management  Study 

Aosenblum,  0.  E. 

Examines  military  training  as  a  total  system  composed  of 
foir  major  subsystems:  recruiting  and  AFEES,  recruit 
training,  specialized  ski  1 1  training,  and  unit  training. 
Overviem  of  toy,  Army,  Narine  Corps,  and  Air  Force 
training.  Presents  Information  an  use  of  SQT  and  ARTEP. 

Seq: 

33 

ID  Number: 

ARI -AM-54-42;  AD-6351  314 

Dote: 

Dec  1354 

Title: 

Combat  Efficiency  Ratings  based  on  30-50  Days  of  Observation 

Author: 

King,  8.  H. 

Corporate  Author: 

AO  Office 

Summary: 

Attempts  to  determine  the  degree  of  similarity  betmeen  short 
term  <30-53  days)  observation  reports  of  combat  performance 
and  those  based  on  DO  days  or  more. 

Key  Points: 

Found  no  significant  difference  betmeen  30-53  observation 
and  longer  period. 

A-16 


8VN0PSE8 


8*q: 

10  Number: 

Data: 

Tltlt: 

Author: 

Corporate  Author: 
Summary: 


Kay  Points: 


34 

ARI-Tft-303;  AD-A077  830 
Jul  1070 

Combat  Oparotions  Training  Effectiveness  Analysis  hods  I : 

1070  Perspective 
Hod  I  in,  8.  n. 

mi 

Developed  a  model  for  a  criterion  raforancad  system  for 
•valuation  of  unit  tactical  performance.  Usas  flRTEP 
•valuation  systoa  as  a  starting  point. 

Hodol  provides:  realistic  simulated  combat  environment  in 
■hi eh  objective  performance  data  cai  be  obtained,  procedures 
for  defining  standerds,  and  techniques  by  ehich  tng 
deficiencies  and  tng/coebat  readiness  can  be  assessed. 
Presents  flRTEP  eeaknesses. 


8eq: 

ID  h umber: 

Date: 

Title: 

Author: 

Corporate  Author: 
Summary: 

Key  Points: 


35 

AD-A104  708 
?  June  1085 

Coeeand  Information  Requirements  of  the  Airland  Battlefield 
(taster's  Thesis  for  US  Army  CtOSC) 

Schmoder,  J.  A. 

USACQSC 

taalysls  of  the  critical  information  requirements  needed  by 
the  Force  Coemander  to  execute  Airland  Battle  doctrine. 
Review  results  of  past  efforts  to  Identify  decision  eefcar 
Key  Information  needs.  Points  out  that  automation  Is  the 
answer  to  providing  timely  and  accurate  information  to 
commanders.  Looks  at  85  Key  elements  of  information. 


Seq: 

ID  Humber: 

Oate: 

Title: 

Author: 

Corporate  Author: 
Summary: 


Key  Points: 


30 

tat  Aval  table 
tat  Aval  table 

Comparison  of  Training  Transfer  and  Effectiveness  fade  Is 
Knerr,  C.  H.  8  tadlw>,  L. 

mi 

taalyzes  and  coepares  training  transfer  and  effectiveness 
models  on  objectives,  components,  units  of  analysis, 
metrics,  and  development. 

Host  of  the  models  are  prescriptive,  rather  than  predictive 
of  effectiveness. 


A-17 


8VNQP9ES 

8eq: 

37 

ID  H iabr: 

ARI-TA-78-A20;  flD-056  791 

Dot*: 

Juno  1979 

Tltlt: 

Consideration  of  A'ty  Training  Oevice  Prof ic fancy  Asstssttnt 

Capabilitits 

Author: 

She 1 nut t,  J.,  Stlllit,  A.,  t  Bercos,  J. 

Corporatt  Author: 

Litton  Hal  Ionics 

Summary: 

Looks  at  salactad  Movy,  Air  Foret  ft  Coast  Guard  programs  to 
illustratt  ganaral  trends  in  the  ust  of  si  tula  tors  for 
evaluation. 

Key  Points: 

Describes  the  fray  Training  and  Evaluation  Systtt.  Although 
it  toy  bt  outdated,  it  provides  a  good  description  of  ehat 

SQTs  aid  ARTEPs  art  all  (±>out . 

8tq: 

39 

ID  Mumbw*: 

AAI-AH-57-30;  flD-9951  909 

Oats: 

1957 

Tltlt: 

Construction  of  Achievetent  and  Ptrfortanct  Tests 

Author: 

Berkhouse,  A.,  Bomstein,  H.,  Broen,  E.,  t  Dub  In,  8. 

Corporatt  Author: 

AG  Office 

Summary: 

Manual  provides  instructions  for  development  and  ust  of 
achievetent  tests. 

Kay  Points: 

Prepared  for  ailitary  instructors  in  Arty  schools  and 
training  centers.  Test  developer  is  taken  step  by  step 
through  process  of  developing  a  test. 

Stq: 

39 

ID  Mimber: 

ARI-TR-328;  AD-A127  943 

Data: 

Feb  1981 

Tltlt: 

Cost  and  Information  Effectiveness  toalysis  (CIEA):  A 

Methodology  for  Evaluating  a  Training  Device  Operational 

Aeadiness  Assessment  Capability 

Author: 

Hatley,  J.,  &  Daedy,  E. 

Corporatt  Author: 

Applied  Science  Associates,  Inc. 

Suttary: 

Presents  problems  encountered  in  the  development  of  DOAAC 

Kty  Points: 

Application  to  training  devices,  i.e.  deciding  betmeen 
design  options.  To  maintain  hi?)  level  of  combat  readiness 
there  must  be  frequent  evaluations  of  Indiv.  and  unit 
proficiency  and  a  means  of  quickly  diagnosing  and 
remediating  performvtce  problems. 

A-18 


Seq: 

ID  N tabor: 

Dot e: 

Till*: 

Author: 

Corpora to  Author: 
Suemary: 

Koy  Points: 


40 

ARI-AP-81-1;  AD-A101  069 

July  1080  _ 

Cost  and  Training  Effectiveness  Aialyels  <CTEA>  Performance 
Quids 

Hot  lick.  A.,  Borgor,  0.,  6  Aomt,  A. 

Litton  hoi  Ionics 

Provides  procadirol  guidance  to  Cost  end  Trolning 
Effectiveness  Analysis  (CTEA). 

flothod  for  Mosuring  and  assossing  cost/af  fact  I  vsness  of 
Aray  tmg  systems.  Providas  dototlod  revise  of  TEEM  <Tmg 
Eff icitncy  Estimation  hodol ),  DIUAD  Dun,  Analogous  Task 
flothod,  and  TAAIHUICE  Ptl  TAA0E. 


S«q: 

ID  ffcaber: 

Data: 

Till*: 

Author: 

Corporate  Author: 
Summary: 


Ksy  Points: 


Seq: 

10  ftaber: 

Oats. 

Title: 


Author: 

Corporate  Author: 
Summary: 


Koy  Points: 


41 

AD-A017  722 
Sop  1079 

Course  Outline:  Instruction  for  Uhit  Trainers  in  Hoe  to 

Conduct  Performance  Training 

Osborn,  U.  C.,  Ford,  J.  P.,  6  Hoon,  H.  L. 

HueAAO 

Detailed  outline  for  a  ten-hour  block  of  Instruction 
designed  to  teach  officers  and  NCQs  hoe  to  nonage  and 
conduct  performance-oriented  training  in  their  units. 
Emphasis  on  performance  oriented  training. 


42 

ARI-AN-82-21;  A0-A127  021 
Jan  1062 

Cree  Performance  Acquirements  for  Emerging  frtor  Weapon 
Systems:  Studies  of  Cree  Size  and  Methods  of  Forecasting 
Human  Factors 

Caepbell,  A.,  Taylor,  E.,  t  Caepbell,  C. 

HueAAO 

Emphasizes  Amy's  major  goals:  readiness,  modernization,  and 
sustaincfcl  I  ity.  Reports  teo  studies  pertaining  to 
eartpoesr  required  to  operate  light  melght  amor  combat 
vehicles  and  methods  of  forecasting  performance 
requirements. 

Lack  of  good  performance  data  eedioned  the  research  and  led 
to  a  mi  lance  on  questionnaire  data.  Presents  Operational 
Sequence  Analysis  <0SA>. 


A-19 


8VH0P8E8 


Seq: 

ID  tUNr: 

Data: 

Title: 

Author: 

Corporate  Author: 
Sueaary: 


Key  Points: 


43 

PRl-RR-ia06;  A0-A1B6  703 
Jan  1089 

Critarion  Performance  Heasirms  for  A1  Tank  Driuor  Tests 
Burroughs,  8.  L. 

AAI 

Provides  criterion  performance  eeosures  for  reliable  tests 
of  non-procedural  HI  tank  driver  skills  that  could  serve  as 
stds  for  talk  driver  Simula  tor  training.  Study  cooparms 
performance  scores  of  HI  tank  drivers  to  novice  Olivers 
performance  scores. 

Based  on  results,  concludes  that  criterion-based  tests  are 
potentially  reliable  quantitative  instruments  for  eeonring 
performance. 


Seq: 

ID  Nueber: 

Date: 

Title: 

Author: 

Corporate  Author: 
Summary: 


Key  Points: 


44 

flRI-flH-79-0;  AD-AQ76  787 
Aug  1079 

Criterion  Performance  Neastrms  of  Leadership  and  Uhl t 
Effectiveness  In  Seal  I  Coabat  Units. 

Doenoy,  A.  0.,  Duffy,  P.  J.,  S  Shlflett,  8. 

ARI 

Gathers  evaluative  data  on  the  processes  and  outcomes  of 
field  training  exercises  from  detachment  eemh ert  and  three 
soirees  external  to  the  detachment. 

Factor  analysis  indicated  major  difference  aas  that  the 
external  sources  viesed  performance  as  multi-dieensional  and 
the  detachment  member s  viesed  performance  as 
uni-dieensional . 


Seq: 

ID  Nueber: 

Date: 

Title: 

Author: 

Corporate  Author: 
Suamary: 


49 

ARI-flR-1103;  flD-AOSS  064 
Feb  1078 

Criterion-Referenced  Job  Proficiency  Testing:  A  Large  Scale 
Application 

Haler,  A.  H.,  S  Hlrshfeld,  S.  F. 

AAI 

SQTs  content  must  be  based  on  systematic  analysis  of  job 
requirements.  Feedback  loop  must  be  mode  to  training 


Key  Points: 


Discusses  the  8QT  program,  its  principles  of  test 
construction,  and  the  benefits  expected  in  Its  utilisation. 


A- 20 


tVNOMB 

taq: 

10  N tabor: 

Data: 

Title 


Author: 

Corporate  Author: 


Kay  Paints: 


46 

AAHW-87-21;  AD-A077  968 
Sap  1976 

Criterion-Referenced  Systaa  Jfeprooch  to  Evaluation  of  Combat 
Units 

Hlrabolla,  A. 

AAI 

Praposas  development  of  adequate  systaa  of  evoluatlon  that 
looks  Ilka  tho  angogaaont  simulation  test  bad.  Stresses 
laportanca  of  dofining  aaafuraaant  concepts,  data  procasslng 
concepts,  and  data  interpretation  concepts. 

Orta  of  tha  philosophical  problems  of  ARTEP  Is  that  it  doas 
not  adaquataly  distinguish  bo  to  son  evaluation  for  training 
diagnosis  and  evaluation  for  accountability. 


8aq: 

ID  Itab*: 

Dote: 

Title: 


Author: 

Corporate  Author: 


Kay  Points: 


47 

flR I -TP-306;  AD-A061  960 
Aug  1978 

Critarion-Aafarancad  Tasting:  A  Critical  Analysis  of 
6a (acted  bode  Is 

Steinheiser,  F.  H.,  (  Epstein,  K.  I. 

Maryland  Uniw 

Revises  folloaing  aodals:  Block,  Crahan,  Eerick,  Dayton  ( 
Mocrmady,  Krieeoll  S  Mi  I  loan  (binomial),  Novlck  (Bayesian), 
Rasch  (logistic),  and  classical  regression. 

Problems:  1)  oonaruenca  between  CRT  oerformance  and 
rmaf-morld  requirements  2)  statistical  Inferences  applied  to 
observed  scores. 


Beq: 

ID  Huaber: 

Date: 

Title: 


Author: 

Corporate  Author: 


Key  Points: 


48 

ARI-m-75-11;  AD-A076  789 
Dec  1975 

Criterion-Referenced  Testing:  A  Discussion  of  Theory  and 
Practice  in  the  Army 

Seezey,  R.  U.,  Pear  I*  tain  R.  B.,  8  Ton,  U.  H. 

AAI 

Rev  leas  literature  on  cri  tar  ion-referenced  testing  (CRT)  and 
the  status  of  criterion-referenced  test  construction  and 
application.  Presents  manual  for  developing  CRTs. 

Discusses  appropriateness  and  controversy  of  eapirical 
estimations  of  CRT  rei lability  and  validity.  Presents 
detailed  revise  of  CRT  theory  and  applications. 


A-21 


fuyMMfA 

ivnumv 

faq: 

10  Mute: 

Oats: 

Title: 


Author: 

Corporate  Author: 


Kay  Points: 


40 

AAI-TA-447;  AD-A005  662 
Jvt  I960 

Crass-Ual  i  dot  ion  of  Predictor  Equations  for  Amor  Cram an 

(to  it  land.  A.,  Eaton,  N.,  t  Naff,  J. 

AAI 

Cross-validation  of  amor  ere— an  porfomanca  predictor 
aquations.  Usa  ASUAB  attest  scorns  as  predictor  measures. 
Attaapt  to  dotamina  if  those  predictors  oouid 
cross-va!  I  date  to  a  nae,  larger  saapla  of  amor  trainees. 
Predictors  earn  fotnd  to  be  valid  for  driver  and 
gumer/looder  parforeanca  at  and  of  training  and  successful 
In  cost  portions  of  the  criterion  eaasures  for  former 
trainees  sho  earn  retested.  Nixed  results  oere  obtained  oith 


8eq: 

10  limber: 

Date: 

Title: 

Author: 

Corporate  Author: 
Stmmary: 


Key  Points: 


90 

AD-A1  It  381 
Nov  1981 

Decision  Hcfclng:  An  Interdlscipl  Inary  Inquiry 
Ungson,  G. 

Organizational  Effectiveness  Aesearch 

Contains  all  papers  and  commentaries  presented  at  conference 

of  decision  making  held  in  Harch  1081. 

Presents  some  relevant  papers  on  decision  mAing.  Focuses 
on  decision  making  theories. 


Seq: 

10  Number: 


Date: 

Title: 

Author: 

Corporate  Author: 


Key  Points: 


51 

ARI-AR-1324;  A0-A113  703 
*>rll  1981 

Description  of  the  ARI  Cram  Performance  Node  I 
Smalm,  A.,  Crumley,  L.,  6  Coke,  J. 

ARI 

Computer-based  model  to  simulate  the  speed  of  performance 
of  cress  varying  in  size  and/or  task  assignments. 
Performance  measurements  used  to  evaluate  the  speed  and 
relative  efficiency  of  cress  varying  in  size  or  structure. 
Computer-based  performance  model . 


A- 2  2 


SYNOPSES 


8aq: 

10  Nato: 

Oats: 

Tills: 

Author: 

Corporal*  Author: 
Smoary: 


Kay  Points: 


52 

ARI-AN-81-11;  A0-A125  805 
Sap  1060 

Oasign  and  Deveiopeent  of  Diognostic  hoostres  for  feoor 
Craaaon  P«»for*ance  -  XH1 
Caepbell,  A.  C. 

HuaAAO 


Oovalop  diognostic  aaosuras  of  individual  and  eras  d»iils 
raqulrad  in  porforoanca  of  XH1  ginnery. 

Prasants  axaeplas  of  appl  i cations  of  systea  porforoanca 
aaasura  concept.  Provides  individual  end  eras  •valuation 
package . 


Soq: 

10  hrnber: 

Data: 

Title: 


Author: 

Corporate  Author: 
Sumary: 


Key  Points: 


S3 

ARI-fft-82-13;  AD-R12?  062 
April  1061 

Oavelop  and  Evaluate  Nee  Training  and  Perforoance  Systaos 
for  haintananca  Job  Evaluation:  Findings,  Plans,  and 
Exaeples 

Harper,  U.,  4  Gutaan,  J. 

AMACAPA  Sciences 

Describes  final  year  of  a  three-year  project  to  develop, 
inpleaent,  and  evaluate  an  ftroy  haintananca  Porforoanca 
Systea  (HPS). 

Provides  specific  oaintanance  related  perforoance  oeasuras. 
Prasants  Return  on  Invastoant  (R0I )  $  figure  and  results  of 
nee  training  and  porforoanca  systea. 


Saq: 

ID  hrnber: 

Data: 

Title: 


Author: 

Corporate  Author: 
Smeary: 


Kay  Points: 


34 

HTEC-70-0-0 105-1;  AD-A106  224 
June  1061 

Oavalopoant  of  the  Autooatad  Perforoance  Assessoant  and 
Reoedial  Training  Systea  (APARTS):  A  Landing  Signal  Officer 
Training  Aid. 

Oreidenbach,  8.,  (  Brictson,  C. 

Dunlap  and  Associates  _ 

Describes  developoent  of  APARTS,  an  autooatad  training  aid 
designed  to  assist  the  Landing  Signal  Officer  <LS0>  in 
training  pilots  on  carrier  landing  tasks. 

Atteept  to  optimize  simulator  utilization  based  on 
perforoance  eoasureoent.  Applies  autooatad  md  integrated 
training  approach  to  ieprov*  night  carrier  landing  throufpi 
•ore  effective  utilization  of  training  devices. 


A- 2  3 


auynMCt 

9TllrX8 

8iq: 

10  Hu 

Oata: 

Title: 


Author: 

Corporate  Author: 
Suaeary: 


Kay  Points: 


59 

AA I -AN-84- 143;  AD-A150  089 
Doe  1864 

Developing  a  Field  frti  I  lory  Training  Systoo  Basod  on 
Devices  and  Sioulations:  Definition  of  the  Gunnery  Teas 
Trainer 

Bishop,  E.  U. ,  Blooo,  A.  F.,  t  Haoliton,  J.  U. 

Dunlap  and  Associates  East,  Inc. 

Dove  I  op  sent  of  a  concept  and  functional  description  of  the 
Gunnery  Teas  Trainer  (GTT) 

Bevies  of  Field  Arty  tmg  devices  indicated  they  sere 
designed  to  provide  practice  but  eith  little,  if  any, 
definition  of  the  task  to  be  learned  or  of  hoe  to  oooyiro 
tmg  success.  GTT  provides  this,  heart  to  be  used  as  a 
readiness  eeasuring  device. 


Sea: 

10  ffcober: 

Dote: 

Title: 


Author: 

Corporate  Author: 
Sineary: 


Key  Points: 


56 

AA I -AN-64- 137;  AD-A150  365 
Oec  1864 

Developing  a  Field  Artillery  Training  Systee  Based  on 
Devices  and  Sioulations:  Evaluation  of  Training  Devices  6 
Sioulations 

Bishop,  E.  U.,  Blooo,  A.,  t  Haoliton,  d. 

Dunlap  t  Assoc.  East,  Inc. 

Prograo  to  develop  the  description  of  a  systoo  for  unit 
level  training  In  the  field  artillery.  Special 
consideration  given  to  the  use  of  training  devices  and 
sioulations. 

Looks  at  existing  and  planned  FA  training  technology,  e  g. 
firing  battery  trainer  (FBT),  battery  cooputer  systoo 
interface  training  siaulator,  NILES,  end  FI8T/F0  Interactive 
video  disk  trainer. 


Seq: 

ID  NuOber : 

Date: 

Title: 

Author: 

Corporate  Author: 
Suawry: 


Key  Points: 


57 

flRI-AN-82-22;  AD-A127  077 
Nay  1860 

Developoent  and  Evaluation  of  a  General  izcfele  dob 
Proficiency  Hatrix  (OJPN) 

Hanson,  T.,  Beho,  A.,  Johnson,  C,  Hishfield,  J,  t  Jests  I  g, 

A . 

Personnel  Decisions  Research  Institute 
GJPN  identifies  coooonai ities  aoong  tasks  sithin  and  across 
NOSs  based  on  behavioral  content.  It  is  used  to  develop 
prioritized  task  lists,  identify  perforeance  oeosvres  for 
critical  tasks,  and  identify  coeean  behavioral  eleeents 
across  tasks. 

Can  be  applied  to  and  facilitate  the  developoent  of  SQTs. 


A-24 


ni  mnocrfi 

SYnUraU 

8eq: 

SB 

ID  Hueber: 

A0-A133  160 

Date: 

March  1862 

Title: 

Deuel opeent  and  Field  Trial  of  a  Syst—  for  Evaluating  the 

Effectiven— s  and  Efficiency  of  a  Training  Progr— 

Author: 

Uitsor,  6.,  t  Kristiansen,  0. 

Corporate  Author: 

AAI,  Fort  Knox  Field  Uhit 

Sueeary : 

Describes  the  develop— nt  and  field  trial  of  the  Training 

Plan  Evaluation  ohich  identifies  specific  training  prop*— 
deficiencies  and  recoeeendg  coirs—  of  action.  Aeport 
assesses  the  use  of  the  TPE  In  evaluating  transition 
training  froe  H60  to  HI. 

Key  Points: 

Assess—  use  of  TPE  In  evaluating  transition  froe  H60  to  HI 
dirlng  0T- III.  Presents  process  (for—  >  used  In  training 
effectiveness  study. 

ID  Nu— er:  ARI-TR-79-94;  AD-A075  431 

Dale:  Itay  1070 

Title:  Bout  I  op— nt  and  laplo— ntation  of  a  porfor—nco  bo— d 

training  — aluation  syst—  for  tht  coebat  arms 

Author:  Bialik,  H,  H.,  4  Bronna,  H. 

Corporate  Author:  AAI 

Su— ary:  Describes  3rd  yoar  affort  of  a  thr—  yoar  projoct  to  develop 

a  syst—  for  providing  individual  skill  training  in  at 
Infantry  unit. 

Kay  Points:  EOploys  6  dtsign  principles:  tmg  is  ptrforaanci  or  I  in  tod; 

individual  ixad;  decontrol  izid;  records  kept  by  (Mediate 
supervisor;  pit  and  sqd  Idrs  identify  and  recoooend  trng; 
requires  for— I  quality  control . 


8eq:  60 

ID  Hueber:  AAI-AM-62-30;  AD-A130  246 

Date:  Jin  1062 

Title:  Develop— nt  of  a  Methodology  for  Conducting  Training 

Effectiveness  Evaluation  of  Air  Defense  Training,  and 
Abstract  of  TEE  Aelated  Literature 
Author:  Fishbume,  A.,  Aolnick,  8.,  6  Larsen,  J. 

Corporate  Author:  Co  I  span  Corp. 

Suneary:  Describes  develop— nt  of  a  syst—  for  conducting  TEE  on  Arey 

AD  training  and  obstructs  of  TEE  related  1 1  tore  tire. 

Key  Points:  Literature  revi—  points  out  so—  relevant  articles. 


A- 2  5 


ID  Hueber: 

MP1CC-TR-S6-8;  RD-A162  631 

Oata: 

Dk  1685 

Title. 

Development  of  a  computer  ■managed  readiness  assessment 
system 

Author: 

Thode,  II. #  t  Buletza,  P. 

Corporate  Author: 

mne 

Effort  conducted  to  develop  a  readiness  training  assessment 
system  for  fleet  air  recon  squadron  too  <UQ-2>  to  provide 
aeetrata,  timely,  end  efficient  assessments  of  the 
operational  readiness  of  aircrew  personnel. 

Key  Points: 

Presents  readiness  training  systems  consisting  of:  1>  matrix 
of  all  events  that  affect  readiness,  2>  computer -managed 
system  to  enter,  process,  store,  and  produce  readiness 
reports,  and  3)  readiness  training  manual. 

62 

flRi-fltf-ei-28;  AD-A126  143 
Ok  1061 

Development  of  o  User's  Guidebook  for  TRfllHUICE  II 
Seezay,  A.,  6  Event,  R. 

SRI 

Poet— nt«  offort  to  develop  user's  guidebook  for  the 
application  of  a  tranefflr  of  training  model  (TRRIHUICE  ll>. 
Model  designed  to  provide  a  framework  for  estimating  the 
effectiveness  of  trainlno  devices. 

Training  devices  can  be  specifically  designed  to  provide 
instructional  benefits  such  as  ieeediate  feedback, 
reinforcement  for  correct  responses,  measurement  of 
achievement  and  other  positive  features  in  complex  skill 
learning  environment. 


Seq:  63 

ID  Hueber:  flRI-flA-1284;  AD-A115  803 

Date:  Jtne  1076 

Title:  Development  of  CRI  Performance  Measures:  Toe  fire  Tactical 

Data  System 

Author:  Hoyt,  II.  0.,  Butler,  R.  K.,  4  Leuig,  P. 

Corporate  Author:  ARI 

Summary:  Summarizes  development  of  CRI  performance  meaares  in  the 

area  of  TRCFIRE  tactical  data  system.  Proposed  courseware 
consists  of  independent  modular  blocks  of  instruction  and  10 
performance  based  module  pre-  and  post-tests. 

Key  Points:  Rtteept  to  implement  CRI  performance  measures. 


A-27 


rmurnu 

•*Q’> 

10  Huber: 
Data: 

Tlilt: 


Author: 

Corporate  Author: 


Kay  Points: 


67 

ARI-flh-80-22;  AD-A128  349 
Oct  1979 

Effects  of  Loader  Transition  on  Unit  Performance: 

Evaluation  of  the  COHTAAIN  Transition  Guide 
Hiller,  ft.  L. 

HutfAQ 

Examines  effectiveness  of  a  program  of  structured  Interviees 
(COHTAAIN)  eith  battalion  resource  personnel  in  facilitating 
transition  in  come and  at  the  company  level. 

Ho  differences  found  beteeen  coemanders  mho  received 
COHTfiflIN  program  and  those  oho  did  not. 


8eq: 

ID  Hueber: 


Date: 

Title: 

Author: 

Corporate  Author: 


Key  Points: 


66 

AAI -AH -64-99;  AD-A136  264 

Feb  1964 

Effects  of  Performance  Feedback  In  Organizational  Setting 
llgen,  0.,  Dugon,  B.,  Hat  tee,  H. ,  Fisher,  C.,  t  Taylor,  8. 
Purdue  Uhivorsity 

Investigates  model  of  performance  feedback  ehich  describes 
the  effects  of  various  dimensions  of  feedback  on 
psychological  processes  and  behavior. 

Points  out  relationship  beteeen  actual  feedback  and 
perceptions  of  feedback. 


8oq: 

ID  Huber: 

Date: 

Title: 

Author: 

Corporate  Author: 
Summary: 


Key  Points: 


69 

ARI-TR-485;  AD-A100  974 
Aug  1980 

Effects  of  Repeated  Engagement  Simulation  Exercises  on 
Individual  and  Collective  Performance. 

Sulzen,  R.  H. 

AAI 

Research  conducted  to  measure  individual  and  tactical 
performance  in  a  series  of  simulation  exercises. 

Results  Indicate  that  collective  tactical  performance  is 
Improved  by  repeated  engagement  sloutatlon  exercises. 


A-20 
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SVN0P8ES 

S*Q: 

73 

10  H imbur  : 

AAI-AA-67-16;  AD-A076  818 

Oats: 

July  1878 

Title: 

Evaluation  of  the  Effectiveness  of  Training  Oevioes: 

Elaboration  and  Application  of  the  Predictive  Aodel 

Author: 

Itwaton,  0.  A.,  Fingerman,  ,  Aose,  A.,  t  Leonard,  A. 

Corporate  Author: 

AIR 

Presents  prel  ieinary  mork  in  developing  and  evoluoting  a 
eodel  ehich  can  be  used  to  predict  and  evaluate  the 
effectiveness  of  training  dift'ices.  Eophasis  placed  on 
transfer  of  training  as  standard  of  effectiveness. 

Keg  Points: 

Presents  predictive  eodel  ehich  odheres  to  certain 
principles  based  on  type  of  task  to  be  trained.  Looks  at 
stimulus,  response,  and  feedbock.  Suggested  feedback:  KOR, 

re  i  n  fore  spent,  automatic  systee  performance  feedback,  etc.  { 

Seq: 

ID  Nueber : 

Date: 

Title: 

Author: 

Corporate  Author: 
Summary: 

Key  Points: 

74 

ARI-TR-76-R2;  AD-A040  911 

Oct  1878 

Evaluation  of  the  Effectiveness  of  Training  Devices: 

Validation  of  a  Predictive  Aodel 

Uheaton,  0.  R.,  Aose,  A.  A.,  8  Fingorean,  P.  U.  9 

Rear  lean  Insitutes  for  Aesearch 

Describes  effort  to  develop  and  validate  a  trmsfer  of 

training  eodel  used  to  predict  effectiveness  of  frey  -  --  i 

training  devices. 

Prel  ieinary  transfer  of  training  eodel  ehich  deals  elth  3 
eajor  classes  of  variables:  appropriateness  (e.g. , 
sieilarity,  criticality,  coeeonotity);  efficiency  (learning 
deficit  and  training  principles  end  techniques);  and 
effectiveness. 

Seq: 

75 

ID  It atm': 

AR1 -APR-79-4;  RD-A076  635 

Oate: 

Her  1878 

Title: 

Evaluator  Attitudes  Toaard  T-TOE  and  H-TOE  Unit  S  true  tires 
in  the  Aanewer  Battalion  Phase  of  the  Aas true turing  of  the 

Heavy  Oivision  Test 

Author: 

Seuts,  E.  A.,  8  Actkinson,  T.  R. 

Corporate  Author: 

ARI 

Susmary: 

Determine  extent  to  ehich  pretest  (pre-trial)  attitudes  of 
evaluators  affected  their  ratings  of  the  norool  TOE  (table 
of  organization  and  equipment)  structures  ehich  mere  tested 
in  the  Maneuver  Phase  of  the  test. 

Key  Points: 

Aid  to  determine  validity  of  evaluator  ratings. 

A- 30 


SYNOPSES 


Saq; 

10  lii jeber: 

Oat*: 

Title: 

Author: 

Corporate  Author: 
Sueeary: 


Kay  Points: 


76 

mi-TA-438;  RD-A060  264 
Apr  1060 

Evaluator  Bating  of  Uhit  Performance  in  Flald  Exarc  isos:  A 
hul tdiaansionol  scaling  analysis 
Modi  in,  8.  N.,  6  Thompson,  P. 

mi 

splits  statistical  analysis  techniques  to  expert  judgsants 
to  explore  systasatic  sathods  to  incorporate  sxpart  silltory 
opinion  into  avoluation  procedures. 

Used  aul ti-dieensional  scaling  techniques  to  determine  hoa 
•any  disansions  the  judges  used  to  evaluate  unit 
perforMnee.  Aesults  indicated  judges  use  only  three 
d leans ions  and  doe  Inant  d leans ion  is  quality  of  overall 
performance. 


Sea: 

ID  Nuaber : 

Date: 

Title: 

Author: 

Corporate  Author: 
Stemary: 


Key  Points: 


77 

Not  Available 
Not  Available 

Examining  the  Link  Bateman  Training  Evaluation  and  Job 
Performance  Criterion  Development 
Hedge,  J.,  Ballantine,  A.,  6  Gould,  B. 

AFHAL  m P  Division 

Points  out  that  eost  research  is  focused  on  predictor 
development  *  not  on  criterion  development.  Air  Force 
coeaunlty  needs  to  spend  more  time  In  developing  and 
utilizing  criterion  measures.  Looks  at  AFHALs  atteept  to 
overcome  this. 

Attempts  to  overcome  tack  of  criterion  measures  by  employing 
a  variety  of  measurement  techniques:  Walkthrough 
Performance  Testing,  task  ratings,  dimensional  ratings, 
global  ratings,  and  Air  Force  aide  ratings. 


Seq: 

ID  Nueber: 

Data: 

Title: 

Author: 

Corporate  Author: 
Summary: 


Key  Points: 


78 

ABI-m-77-8;  PD-A077  925 
Dec  7? 

Expert  Infantryman  Squad  and  Platoon  Evaluation  (EISPE)  I 
Concept:  Evaluation  and  Observations 
Strasel,  H.  C.,  Ayan,  T.  0.,  6  Word,  L. 
mi 

8th  Inf  Div’s  attempt  to  change  focus  from  eemory  of 
sequential  processes  to  realistic  performance.  Feedback  and 
post-test  training  period  should  be  incorporated. 

Evaluation  checklists  should  be  aimed  at  objective  terminal 
performance  not  just  evaluations  of  process  leading  to  that 
performance. 


A- 31 


* 

SWSfw 

Seq: 

79 

ID  (Sober: 

ARI-AH-77-8;  AD-A07?  928 

Data: 

8ep  1977 

Title: 

E^ort  Infantryean  Squad  and  Platoon  Evaluation  (EiSPE)  II 

Concept,  Report  of  Exercise  Observations 

Author: 

Ayan,  T.  0. 

CorporaU  Author: 

ARI 

Scenery:  Tactical  realise  enhances  adequacy  of  tost. 

Kay  Paint*:  To  inhere*  us*  as  training  vehicle,  there  should  bt  a  r*p*at 

of  portions  not  perforeed  satisfactorily  first  tie*  through. 


Seq: 

80 

ID  Hiobo*: 

Hot  Available 

OaU: 

Hot  Available 

Title: 

Facilitating  and  Hindering  Factors  In  lepleeentlng 

Hanagerlal  Technology:  A  Soclo-Technlcal  Systee  Process 

Author: 

Salas,  E. 

CorporaU  Author: 

Hot  Available 

Sueeary: 

Study  designed  to  discover  through  exaei nation  of 
decision-eaking  processes  of  eanagers  shat  factors 

facilitaU  or  hinder  the  lapleeentatlon  of  huean  resources 
eanageeent  technologies. 

Key  PoinU: 

Application  of  policy  capturing  techniques. 

Seq:  81 

10  Hu*w:  HAHI-1050;  AD-A661  794 

Dot*:  Oct  I960 

T 1 1 lot:  Factor  Analysis  of  Aviation  Training  Measures  and  Post 

Training  Parforacre*  Evaluations 

Author:  Booth,  A.  F.,  t  Berkshire,  J.  A. 

Corporal*  Author:  Haul  Aerospace  hod  leal  Institute 

Si„— ary :  Study  conducted  to  d* Ursine  ihethir  factors  (previously 

Identified  in  “factor  structure"  of  naval  air  training 
variables)  are  represented  in  training  records  of  another 
pilot  seep  I*.  Also  relates  factors  to  perforecre*  after 
graduation  free  training. 

Key  Points:  {^plication  of  factor  analysis  to  perforecre*  evaluations. 


A- 3  2 


AUMMAM 

wnunu 

8eq: 

10  H u*er: 

Dot*: 

Tlllt: 


Author: 

Corporate  Author: 
timeory: 


Kty  Points: 


82 

Not  Nuoilablt 
Fob  1878 

Factors  Affecting  Overseas  8ucctss  in  Industry  (Paper 
prtstnttd  at  Society  of  Intercul  tural  Education,  Training, 
and  Aesearch) 

Russel,  P.  JT . ,  t  Dickinson,  T. 

Hot  Available 

Describes  decision  taking  procedure  that  can  be  used  in 
cross-cultural  selection. 

Utilizes  policy  c^turing  to  identify  ioportant  dimensions 
of  overseas  success. 


ID  Itabw': 

Date: 

Title: 

Author: 

Corporate  Author: 
Suatary: 


Key  Points: 


83 

Not  Aval  table 
Not  Available 

Factors  Affecting  the  Selection  of  Amt icon  Managers  for 
taerseas  Assignment 
Russel,  P.  S  Dickinson,  T. 

Not  Available 

Described  and  applied  a  six-step  approach  to  nodel  and  study 
strategies  in  taking  overseas  selection  decisions. 

Utilizes  policy  capturing  to  inderstand  decision  taking 
strategies. 


Seq: 

ID  Nimber: 


Date: 

Title: 


Author: 

Corporate  Author: 


84 

AAI-AB-1351;  AD-A138  335 
Aug  1982 

Feedback  Needs  of  Training  Developers  and  Evaluators 
Uitter,  B.  0.,  t  Burnside,  B.  L. 

AAI  Fort  Knox  Field  (Jhlt 

Report  suggests  feedback  presently  <**ai l<X>le  to  training 
developers  Is  lacking  in  both  specificity  and  objectivity. 
Need  :  better  coordination  beteeen  DTD  and  DOES,  tore  DTD 
contacts  eith  field,  tore  emphasis  on  hands-on  and 
obj  active  testing,  and  of  developing  computer-based  data 
analysis  and  techniques  to  handle  feedback.  Presents  list 
of  available  feedback. 


Key  Points: 


8VN0PSCS 


faq: 

10  N^ir: 

Oote: 

Title: 

Author: 

Corporate  Author: 
Summary: 

Key  Points: 

85 

ARI-TA-78-85;  AD-A056  330 

Feb  1078 

Field  Measurement  and  Data  Collection  Systee  for  Engojumt 
Sieulation  Field  Exercises. 

O'Heeron,  A.  K.,  Hovel  1 ,  U  Y.,  t  Frazier,  T.  U. 

Behauioral  Technology  Consultants, Inc 

Describes  six  systees  for  locating  vehicle  positions,  in 
teres  of  requi reeents  for  engogeeent  sieulation  data 
analysis  and  in  teres  of  usefulness. 

Engogeeent  sieulation  techniques  nom  being  used  to  train 
frmy  coebat  wits  require  better  instrumentation  and  methods 
to  gather  accurate  data  for  unit  evaluation. 

Seq: 

8b 

ID  Number: 

flAI-AA-1323;  AD-A134  388 

Date: 

Aug  1081 

Title: 

Field  Performance  Feedback:  A  Problem  Review 

Author: 

Burnside,  8.  L. 

Corporate  Author: 

DAI 

SuMary: 

halnly  addresses  external  feedback  systee  (floe  of 
information  free  field  units  to  Arey  Centers/Schools). 

Points  to  existing  records  available  at  battallion  level. 

Key  Points: 

Looks  at  feedback  loop  from  units  to  fray  Centers/Schools. 
Looks  at  ARTEP  and  SOT.  flefces  recommendations  on  shat 
type  of  feedback  should  be  provided. 

Seq: 

87 

10  Number: 

ARI-TR-524;  AD-A128  470 

Dote: 

hay  1081 

Title: 

Field  Survey  of  Current  Practices  and  Problems  in  Arey  Unit 
Training  with  Implications  for  Fielding  and  Training  with 
MILES 

Author: 

Gray,  C.,  Clovis,  E.,  Muller,  T.  t  Cuminghae,  R. 

Corporate  Author  : 

Perceptronics 

Sumeary: 

Survey  of  CONUS  active  infantry  and  armor  divisions 
conducted  to  1)  determine  hoe  they  ecroge,  prepare,  and 
conduct  unit  tactical  training  and  2)  find  out  shy 

PEAL  TRAIN /SCOPES  has  not  been  more  eidely  used. 

Key  Points: 

Interviews  and  questionnaires  on:  acceptance  of 
AEALTAAIN/SCQPES  training  methods,  expectations  and  concerns 
re:  NILES,  practices  and  problems  in  unit  training 
management,  etc.  Info  has  served  as  input  to  planing 
frmy-eide  NILES  ieplementation. 

A- 3  4 


SVN0P8E8 


8oq: 

ID  H taber: 

Date: 

Tills: 

Author: 

Corpora to  Author: 
Su— ary: 


Key  Points: 


86 

AAI-TR-680;  AD-AI90  978 
JU>  1989 

Forecasting  Device  Effect i von— s:  Uoi  I.  Issuos 
Rom,  R. ,  Wheaton,  0.,  4  Vat—,  L. 

Amt  icon  Institute  for  Aesearch 

Discuss—  a  nteber  of  issu—  that  b—r  on  tho  dovoiopeont  of 
for— I  analytic  —thods  for  predicting  tho  potential 
effectiveness  of  alternative  training  dovlc— . 

Discussion  covers  theoretical,  practical  and  esthodo logical 
issu—  uncovered  during  rev  I—  of  the  literature. 


Seq: 

10  N— b—: 

Dote: 

Title: 

Author: 

Corporate  Author: 
Su—ary: 


Key  Points: 


89 

RRI-AP-65-25 
June  1985 

Forecasting  Device  Effectiveness:  Uoluee  II.  Procedure 
Rose,  R.,  itsa ton,  6.,  4  Vat—,  L. 

Not  Aval (able 

Presents  interactive  eenu-driven,  coeputer-bosed  eodel 
ehich  device  designer  can  use  to  determine  alternative 
designs.  Looks  at  fotr  criterion  constructs  of  device 
effect.:  tmg  probl— ,  acquisition  effectiveness,  transfer 
probl— ,  tmg  efficiency. 

Training  probl—  construct:  look  at  shat  type  of 
proficiency  is  required,  proficiency  KSA‘s  of  typical 
soldiers  before  using  the  device,  aid  hoe  difficult  far 
train—  to  acquire  required  proficiency. 


Seq: 

ID  Huaber: 

Dote: 

Title: 

Author: 

Corporate  Author: 
8u— ary: 


Key  Points: 


90 

RRI-flN-61-29;  RD-A126  197 
Dec  1981 

Guidebook  for  Users  of  TAAINUICE  II 
Seezey,  R.,  8  Evans,  R. 

API 

Docueents  transfer  of  training  eodel .  Hoin  purpose  of 
TRAINUICE  II:  provide  — thod  for  assessing  training  devic— 
or  training  device  concepts  in  early  d— ign  phas— . 

Six  coeponents  are  coebined  to  derive  index  of  training 
device  effec tiven— s:  coverage  requlr— ent  anal.,  coverage 
anal.,  training  prof icioncy —al .,  learning  difficulty 
anal.,  physical  characteristics  anal.,  functional 
characteristics  anal. 


A- 3  5 


teq: 

ID  Nwbar: 

Data: 

Title: 

Author: 

Corporota  Author: 
Sum  fry: 


Kay  Points: 


91 

flR I -P-77-5;  AD-A055  632 
How  1977 

Handbook  for  tha  Development  of  8k III  Quol iflcation  Tasts 
Osborn,  U. ,  Caapbell,  R.,  6  Ford,  J. 

HuaflAO 

Cowars  both  technical  and  odatnlstrotlwa  procedures  to 
folloa  ahan  preparing  a  f laid  tested  8QT. 

Hoa  SQT  tasts  art  dawalop ad. 


Sea: 

ID  Huabar: 

Data: 

Titla: 

Author: 

Corporota  Author: 
Suainry: 

Kay  Points: 


92 

flR I -flR- 1305;  AD-A128  066 
Dac  1960 

HuMn  Foe  tors  Evaluation  of  Sa  lac  tad  STANO  Dawioas  Eaployad 
in  a  Mochanizad  Infantry  Platoon 
Saootz,  E.  R. 
flR  I 

Hunan  Factors  Ewaluation  of  four  Surval  I  lonca,  Target 
Requisition,  and  Night  Obsarwatlon  daw  I  cos. 

Rasul ts  usad  to  rafina  tactical  doctrina  at  tha  school  and 
to  assist  in  determining  tha  design  of  future  training 
daw ices. 


Seq: 

10  \ 

Data: 

Titla: 


Author: 

Corporate  Author: 
6unaanj* 

Kay  Points: 


93 

Hot  Awai labia 
Oct  1972 

Hunan  Parfomanca  Effactiwanass  and  tha  Systaas  Haasuraaant 
Bad  (Article  in  Journal  of  Applied  Psychology,  wo  I  56,  nr. 
3. 

Uhfanar,  J.  E. 

Journal  of  Applied  Psychology  Uol  56 
Discusses  the  Concept  of  Systaas  Haasuraaant  Bad. 

Stresses  that  aptitudes,  job  deoands,  and  sirrotrxJing 
conditions  coalesce  to  yield  varying  towels  of  performance. 
Looks  at  cognitive  and  non-cognitiwe  varieties  affecting 
per  for  nance 


A-36 


SYMCPSES 


giq; 

10  Number: 

Data: 

Title: 

Author: 

Corporate  Author: 
Summary: 


Key  Points: 


04 

BESflL-TRH-223;  AD-713  463 
Jun  1970 

Humcn  Performance  Experimentation  in  Higptt  Operations: 
Technology  t  instrumentation  for  Field  Aeseorch 
Hyman,  A.,  t  Sternberg,  J.  J. 

Army  Behavior  and  Systems  Aeseorch  LA 
Provides  a  description  and  evaluation  of  instrumentation 
system  ehich  includes  training,  testing,  and  control  methods 
and  procedures  as  aell  as  special  experimental  techniques 
and  instrumentation. 

Instrumentation  system  found  to  aid  greatly  In  the 
collection  of  reliable  and  valid  experimentation  data. 


Seq: 

10  Number: 

Date: 

Title: 

Author: 

Corporate  Author: 
Sueaary: 


Key  Points: 


09 

ARI-flH-79-40;  AD-A079  371 
Sep  1070 

Implementation  and  Evaluation  of  the  Tank  Cram  Training 
Program  for  USAAEUR  units 
Kress,  0.,  S  McGuire,  U.  J. 

Hum 

Compares  previously  developed  ttnk  cree  ginnery  training 
program  (perforeance  oriented  and  simulation  based)  against 
conventional  program. 

Training  proy'ae  effectiveness  assessed  for  both  groups 
based  on  their  cree  gunnery  performance  on  Table  vi .  study 
stresses  need  to  develop  re  I  (able  tank  ginnery  criterion 
performance  standards  and  measurement  techniques. 


Seq: 

10  Number: 

Oate: 

Title: 

Author: 

Corporate  Author: 
Summary: 


Key  Points: 


06 

ARI-flN-83-64;  AD-A136  601 
Sep  1083 

(■plications  for  Development  of  Collective  Training 
Information  System  <CTIS> 

Scott,  T.,  6  Ekstroa,  A. 

ARI 

Presents  results  of  information  needs  assessment  to 
determine  components  of  a  collective  training  information 
system. 

Interview  revealed  definite  need  for  feedback  free  field 
users  of  ARTEP  documents  to  the  developers  pertaining  to  the 
quality  and  utility  of  the  product. 


A-37 


SVN0PSE8 


Seq: 

10  Ntab«r: 

DaU: 

Title: 


Author: 

Corporat*  Author: 
Sumary: 


Koy  Points: 


9? 

ARI-TR-79-R26;  AD-A064  271 

How  1976  _ 

leproved  Arty  Training  and  Eualuation  Proyae  (ARTEP) 
Hothods  for  Unit  Evaluation,  Uol  I:  Exocut  I  vo  Sunoary; 

8tudy  Oaslgn  end  Fiald  Research 

Hovrcn,  A.  0.,  Albert,  0.  0.,  6  HoCui  lough,  T.  J. 

Hcaan  Scioncas  Aaioarch,  Inc  _ 

Analyzes  existing  aathods  of  iaploaonting  ARTEP  for  a 
tank/aach  Infantry  Took  Fores.  Objective:  identify  probleas 
in  conduct  of  ARTEP  for  fiald  mils,  devalop  reeedies,  and 
incorporate  thaa  into  a  practicoi  fiald  guide. 

Study  conducted  ahan  ARTEP  systaa  aas  in  early  stages  of 
dowel opeent.  Various  technical  problees  inherent  In 
training  and  evaluation  outline  (Tt£0)  are  presented. 


Seq: 

ID  Huaber: 

Date: 

Title: 

Author: 

Corporate  Author: 
Suaeary: 


Key  Points: 


ARI-TR-78-A27;  AD-AOW  763 
Nov  1976 

loproved  Army  Training  and  Evaluation  Prograe  (ARTEP) 
Nethods  for  Unit  Evaluation,  vol  II:  Analysis 
Navran,  H.  D.,  Albert,  D.  D.,  4  NcCuiiough,  T.  J. 

•  eJPwl  9CI ViCS#  nlsVQPCii^  inC 

Provides  data  analysis  and  recoeeondations  for  refining 
current  ARTEP  iep  I  aeon  tat  ion. 

Significant  eephasls  oust  be  placed  on  adequate  training 
for  evaluator /control  ler.  Presents  recoeeendotions  for 
fol  loe-on  research,  e.g.  integration  of  nee  technology  into 
ARTEP' s  evaluation  coeponent  (engagaeent  siaulation,  battle 
sieulation,  etc.). 


8eq: 

ID  Huaber: 

Date: 

Title: 

Author: 

Corporate  Author: 
SuMary: 


Key  Points: 


99 

ARI-TA-78-A28;  AD-A064  272 
Nov  1978 

loproved  Amy  Training  and  Evaluation  Prograe  (ARTEP) 
Nethods  for  Unit  Evaluation,  Uol  III:  Field  Guidance 
Hovron,  H.  D.,  Albert,  D.  0.,  I  HcCul  lough,  T.  J. 

Huean  Sciences  Research,  Inc  _ 

Prototype  guide  for  battalion  level  ARTEP.  Based  on 
recoaeandations  presented  in  vol  II. 

Stresses  ARTEP  principles  and  applications.  Points  out  that 
basic  principle  is  performance  oriented  training. 


A-38 


BVMOPBEB 

teq: 

10  M kabor: 

Oats: 

Title: 


Author: 

Corporate  Author: 
BuMory: 


Kay  Points: 


8aq: 

10  frabv: 

Data: 

Titla: 


Author: 

Corporota  Author: 
Buanory: 


KSy  Points: 


8aq: 

ID  Nuaber: 

Dots: 

Titla: 

Author: 

Corporota  Author  : 

fi^MPtl' 

Kay  Points: 


100 

flAI-TR-79-R23;  AD-A079  469 

f^r  1079  _ 

laproved  frog  Training  and  Evaluation  Progran  (rthftP) 
Hothods  for  Uhlt  Evaluation,  Uoi  U:  Analysis  of  Alternative 
Training  Strataglos 

Matron,  H.  0.,  hcfarling,  L.  H.,  t  Hill,  H. 

Husan  Sciences  Rasoareh,  Inc 

Amlyzas  alternative  training  sottings  aval  I  able  to  Bn _ 

training  aanogors  for  tho  conduct  of  training  oithin  flRTEP. 
Battings  includo  convantional  field  exorcise,  ongogaaont 
slaulation  field  ex&xlse,  etc. 

Describes  capability  for  perforaance  neosuronent/dl agnosia 
for  aoch  setting. 


101 

PRI-TA-79-A24;  AD-A075  663 

Apr  1079  _ 

laproved  Aray  Training  and  Evaluation  Prograe  (flRTEP) 
Hothods  for  Unit  Evaluation,  Uoi  Ul:  Conventional  flRTEP 
Missions  and  Engageaent  Biaulations: 

Matron,  H.  0.,  B  Reporting,  L.  H. 

Htaan  Sciences  Research,  Inc 

Exaalnes  4  Issues:  develops  on  t  of  accurate,  cooprehensive 
criteria  and  aeasuras  of  unit  perforaance;  s  true  tire  t 

functions  of  evaluator /control  ter  teas;  _ 

reduct  ion/ integration  of  eng  sie  data  and  flRTEP  data;  use 
of  data  to  establish  tng  objectives. 

Points  out  4  sain  issues  that  bear  on  conduct  of  unit 
proficiency  assesseent. 


102 

RRI-TR-79-R25;  FD-R076  957 

Apr  1979  _ 

laproved  fray  Training  and  Evaluation  Progran  (flRTEP) 
Methods  for  Unit  Evaluation,  Uoi  Ul  I :  Executive  Sunaary 
Matron,  H.  D.,  t  Uanschura,  R.  0. 

Hunan  Sciences  Research,  Inc 
Bueoarizes  previous  6  volunes. 

Major  flRTEP  probleas:  little  tine  for  evaluator  training; 
poor  intonation  of  ratings  for  diagnosis  and  tracking  of 
errors;  attoinistration  incoopatible  aith  established 
principles  of  training;  TtEO  foreat  not  oble  to  idsntify 
specific  errors. 


A-39 


BtIUiu 

teq: 

10  Nuabar : 

Data: 

Title: 


Author: 

Corporata  Author: 
Suaacry: 


Kay  Points: 


8eq: 

10  Huabar: 

Oota: 

Titla: 

Author: 

Corpora ta  Author: 
Staaary: 


Kay  Points: 


Smq: 

ID  Huabar: 

Oota: 

Titla: 

Author: 

Corporata  Author: 
Summary: 


103 

AAI-AP-79-7;  AD-A073  470 

Apr  1079  _ 

Improved  Aray  Training  and  Evaluation  Prograa  (ARTEP) 
Hathods  for  Unit  Evaluation:  Guidance  for  Planning  and 
Conducting  Coapany-level  Fiatd  Exarc  Isas 
Hovron,  H.  0.,  HcCul  lough,  T.  J.,  HcFarling,  L.  4 
MartscNra,  A. 

Hiaan  Scioncos  Rasaarch 

Attaapt  to  satisfy  naads  docuaantad  in  tha  first  study 
phase.  Prasants  a  training  prograa  on  tha  conduct  of  an 
evaluator /control  I  or  school. 

Persistant  problaa  in  fray:  institutional izad  lack  of 
appreciation  for  tha  rola  evaluators  can  and  oust  play  in 
evaluating  field  performance  and  in  acting  as  trainers. 


104 

Not  Aval  labia 
Horch  1983 

laproving  tha  Training  Approach 
Ouncan,  C.  8.,  4  Hartjan,  R.  C. 

HQ  TAADOC  /  Armor  Rogozina 

Army  must:  1)  modify  currant  analysis  approach  (collective 
requirements  to  <fr*iva  indiv  skill  training);  2)  adopt 
proactive  training  approach;  3>  train  for  success  on 
battlefield;  4)  bond  soldiers  to  quality  leaders  in 
cohesive  units. 

Phi losophical  ly,  TAAOOC  understands  shat  to  do  -  the 
question  is  ehether  TAAOOC  leadership  can  influence  the 
schools. 


105 

AD-AIM  738 
Omc  1983 

In  Search  of  Coabat  Readiness  in  the  USHC 
Stcfcl,  P. 

Naval  Postgraduate  School 

toa lysis  of  the  factors  that  acta  a  USNC  unit  coabat  reody. 
Presents  results  of  a  survey  of  46  USNC  officers  based  on  a 
readiness  modal . 

Survey  data  analyzed  using  bootstrap  methodology,  shereby 
quantitative  values  are  derived  from  qualitative  value 
Judgments. 


Key  Points: 


ni  mnifirn 
WnDrScS 


Am* 

10  Nuriw'  . 


DoU: 

Title: 

Author: 


Corporate  Author: 


Kog  Points: 


100 

ARI-AH  70-22;  AD-A077  177 
How  1070 

Initiol  ARTEP  Validation  Results:  1074  -  1075 
Hayes,  d.  F.,  0  Hollis,  H.  A. 

Amt  loan  Institute  for  Bosoarch 
Standards  too  subjective;  evaluator  performance  too 
erratic,  doubtful  that  typical  •valuator  personnel  could  bo 
adequately  trainad  to  M«t  standardisation  regulreMnts;  no 
guidelines  for  adjusting  standards  to  account  for  varying 
tost  conditions. 

Situational  artificiality  of  ton  required  to  achieve 
standardization;  has  negative  ispoct  on  validity  as  a  tmg 
guide;  ARTEP  •valuation  represent*  CUU1IHRTI0H  of  training, 
■hich  should  not  be  the  case. 


8eq: 

ID  Nuaber: 

Date: 

Title: 

Author: 

Corporate  Author: 
SiaHKrg: 

Keg  Points: 


107 

AR I -AH- 05-5;  AD-A150  302 
Jan  1085 

Instructional  ffrprooches  for  Individualizing  Basic  Rifle 
Harkseanship  Training 
Haxey,  J.  1.,  C  Seezey,  A.  U. 

Litton  Heltonies  Sgstees  Dev. 

Presents  literature  reviee  of  educational  and  training 
research  on  Individualized  training. 

Points  to  study  by  Rosen  and  Behringer  oho  found:  skills 
taught  to  eeot  current  stds  but  not  in  the  best  «ay, 
criteria  do  not  Met  current  req’d  coebat  characteristics, 
gap  betoeen  current  stds  t  conditions  and  those  required. 


Seq: 

ID  Nusber: 

Date: 

Title. 

Author: 

Corporate  Author: 
Sueeary: 


Key  Points: 


106 

AD-A14Q  417 
Dec  1064 

Interactive  Graphics  Simulator:  Design,  Development,  and 
Effectiveness/Cost  Evaluation 
Richardson,  J.,  Harmon,  K.,  t  Keller,  R. 

Essex  Corporation 

Design,  developeent,  and  iepleaentation  of  Interactive 
Graphics  Sieulator  (IDS). 

Hethodology  to  detereine  training  and  cost  effectiveness. 


A-41 


SYNOPSES 


Seq: 

10  Haber: 

Oat*: 

Till*: 

Author: 

Corporate  Author: 
Scaaary: 


Kay  Points: 


too 

Not  Auai labia 
1066 

JAN:  A  Tachniqua  far  Analyzing  Grove  Judgment  (Article 
appearing  in  Jotmal  of  Exper imental  Education,  Uol .  4> 
Chrlstol,  A.  E. 

Journal  of  Export non la I  Psychology 

Utilizos  Judgment  Arta  lysis  (JAN)  to  identify  mi  dascriba 
tho  rating  policies  aithin  a  board  of  judges. 

Application  of  JAN  technique. 


Sea: 

ID  Nueber: 

Date: 

Title: 

Author: 

Corporate  Author: 
Summary: 


Key  Points: 


110 

ARI-AP-61-17;  AD-A120  774 
Sep  1061 

Job  Aid  for  Nodlfying  Ineffective  or  Inefficient  Training 

Prograas 

Kristiansen,  0. 

ARI 

Job  aid  ehich  addresses  the  problen  of  hoe  to  aodify 
training  ahen  one  has  conducted  a  Tng  Prograe  Evaluation 
and  found  that  certain  changes  are  needed. 

Points  out  the  proto  lees  that  often  result  free  inefficient 
training  e.g. ,  unecessary  instruction  on  skills  mi 
knoeledgos  the  soldier  already  has  or  ahen  practice  tiee  is 
cut  in  the  naee  of  efficiency. 


Seq: 

ID  Nueber: 

Date: 

Title: 

Author: 

Corporate  Author: 
Summary: 


111 

flFHRL-TP-85-51;  AD-A164  837 
Feb  1086 

Job  Perforaanco  Measurement  Classification  Scheme  for 

Ual idation  Research  in  the  Military 

Kavanogh,  M.,  Borean,  U.,  Hedge,  J.  t  Gould,  R. 

McFann-Oray  Associates,  Inc 

Outlines  the  development  of  a  performance  meastsement 
classification.  Focus  on  job  performance  criterion 
development. 

Model  looks  at  input  variables  (individual  characteristics, 
meosureeent  method)  process  variables  (  cognitive 
processes)  and  outcome  variables  (performance  measurement 
quality).  Performance  measurement  criteria  to  be 
considered  are  suggested. 


Key  Points: 


SYNOPSES 


S**: 

10  Hufcw*: 

Data: 

Title: 

Author: 

Corporata  Author: 
Summary: 


Key  Points: 


112 

ARI-AH-63-32;  AD-A135  §79 
Sap  1963 

Job  Saapta  Tasts  os  Pradictors  of  HI  Quinary  Perforates: 
Appandixas  ft-€ 

Biers,  0.,  t  Sauer,  0. 

Sgstaas  Rasaarch  Loborotorias 

Usad  gptituda  aaosuraaant  aath.  to  design  job  soap  I  a 
tasts  for  armor  craaaan.  Daualopad  sauan  Job  soaple  tasts 
to  ineludo:  thraa  coaputar  based  and  four  H-0  tasts. 
Uarious  job  soap I a  tasts  ora  prasantad.  Aasults  Indicatad 
that  linaar  coabi  not  ions  of  job  soap  I  a  tast  eeastres 
occountad  for  very  high  proportion  of  variability  in  crass 
post  success  at  Annual  Qualifications. 


Saq 

10  Number: 

Data: 

Titla: 

Author: 

Corporata  Author: 
Suaiarg: 


Kag  Points: 


113 

STP  M3E-JB 
Apr  1965 

m  flbroas  Tank  Sgstao  Nechonic  Job  Book,  AOS  63E10/20 
Not  Appliccbla 
US  Aray  Ordnanca  School 

Usad  as  an  NCO  training  aanagoaant  tool  to  record 
demonstrated  proficiency  on  soldier's  aanuol  (all  coaaon 
and  NOS-specific)  tasks  for  shich  the  8L  1  or  2  soldier  is 
responsible.  Provides  space  for  supervisor  to  record  go  or 
no-go  and  data. 

Basic  idea  Is  good,  but  doubtful  that  it  is  utilized  sith 
any  degree  of  effect  iveness. 


Saq: 

ID  Nuaber: 

Data: 

Titla: 

Author: 

Corporata  Author: 
Sumeory: 


Keg  Points: 


114 

RR I -P-78-2 
Feb  1978 

Naintanance  of  Perforaance  Effactluonass 
Horabin,  I.,  Katz,  11.,  t  8hialds,  J. 

ARI 

Emphasis  on  supervisors'  invo  I  vacant  and  on  dealing  alth 
performance  problems  at  the  specific  problee  (aval. 
Oa-taphosizas  training  and  technological  solutions. 

Points  out  need  to  deal  alth  perforaance  problems  at 
specific  level.  Supervisors'  perforaance  crucial  to 
aalntananca  of  affect ivanass.  It  Is  laportont  to 
dataraine  shot  the  objectives  are  and  horn  evaluation  mill 
be  node. 


A-43 


8VN0P8ES 


Seq:  IIS 

10  Hu**:  AAHIN-64-3;  RD-A13?  1?t 

Data:  Jon  1084 

Title:  Maintenance  Performance  System  (Organizational)  Handbook 

for  Certifying  Hochvtics  In  Division  88  Armor  Units 
Author:  Splker,  A.,  Harper,  M.,  t  Uotkyns,  A., 

Corporate  Author:  fhacapa  Sciences,  Inc. 

Summary:  Effort  to  develop  the  Maintenance  Porforeance  Systee 

Organizational  (HPS-O).  MPS-0  is  an  integrated  systee  for 
oeasuring  eaintenance  mechanics.  Looks  at  Certification 
Progress  in  the  Arey,  Navy,  Air  Force,  «td  private 
industry. 

Key  Points:  Presents  effort  to  develop  a  systee  to  eeasure  eaintenance 

certification  porforeance. 


Seq:  118 

10  Nueber:  AAI-AN-B4-2;  AD-A137  571 

Date:  Jan  1984 

Title:  Maintenance  Porforeance  Systee  (Organizational)  Inforeation 

and  Evaluation  Systee  Design  USES)  Design  Considerations 
Author:  Slapson,  H,  Gutean,  J.,  t  Jarosz,  C. 

Corporate  Author:  Anacapa  Sciences 

Summary:  Design  considerations  relating  to  the  ItES  of  the 

Nalntonance  Porforeance  Systee  Organizational . 

Key  Points:  Use  as  evaluation  tool .  Use  as  eanageeent  inforeation 

feedback  systee.  Feedback  provided  to  fray  coaeandors, 
eanogers,  trainers  to  alloe  thee  to  revise 
eaintenance/troining  performance. 


Seq: 

117 

ID  ffcetoer: 

Not  Available 

Oats: 

1983 

Title: 

Neasures  of  Coapansatory  and  Noncoepensatory  Nodels  of 

Decision  Behavior:  Process  Tracing  versus  Policy  Capturing 
(Article  eppoaring  in  Organizational  Bohcnn'or  and  Hunan 

Performance,  Uol.  31) 

Author: 

Billings,  A.,  t  Marcus,  8. 

Corporate  Author: 

Organization  Behavior  and  Human  P erf. 

Steeary: 

Examines  the  validity  of  policy  capturing  and  process 
tracing  measures  of  decision  making. 

Key  Points: 

Application  of  AMOUR  and  information  board  technique 
(linear  regression  commonly  used)  to  policy  captiring 
technique. 

A-44 


SVM0P8E8 

Uq: 

ID  H labor: 

Dot*: 

Title: 


Author: 

Corporate  Author: 
Suaaory: 

Kay  Points: 


Sea: 

ID  Nuabar: 

Oota: 

Titla: 


Author: 

Corporota  Author: 
Siaeary: 


Kay  Points: 


Saq: 

10  Nuabar: 

Data: 

Titla: 

Author: 

Corporota  Author: 
Suaeary: 


Kay  Points: 


118 

AD-A063  081 
8apt  1978 

Hathod  to  Oataraina  Divisional  Enginoar  Battalions  Training 
Noasuras  of  Effectiveness  (Hester's  thesis,  Novel 
Postgroduata  School ) 

Gibson,  L.  P.,  JT. 

Nova  I  Postgroduata  School 

Presents  aathodology  to  dataraina  tha  training  oaasuros  of 
effectiveness  for  divisional  engineer  battallions. 

Looks  at  training  eeasires  of  effectiveness. 


119 

Not  Aval  let  la 
Nov  1979 

Aathodology  to  develop  the  Criteria  and  Criteria  Weightings 
for  Assessing  Sifcinl  t  Effectiveness  in  Organizations 
(Article  appearing  in  Acodeay  of  Aanageoent  Jotmal ,  vo) 

22) 

Hitt,  H.,  t  Aiddleaist,  0. 
ncoony  or  nonogaMm  journal 

Study  offars  and  tests  cat  iaproved  procedure  for  aeasuring 
organizational  sibunit  effectiveness  by  isolating  relevant 
criteria  and  deters  in  ing  criteria  eelghts  ei  thin  a  large 

coop  I  ex  organization.  - 

Use  of  policy  cap  Ur  ing  at  a  state  health  departeent. 


120 

ARI-TB-569;  HD-A13I  969 
Aarch  1982 

Aethods  of  Evaluating  Tank  Platoon  Battle  Run  Performance: 
Design  Guide! ines 

Allen,  T.,  Johnson,  E.,  Ill,  Knerr,  A.  t  Boycan,  G. 

HR  I 

Provides  guidelines  to  assist  local  battalions  in  the  tasks 
of  planning,  conducting,  and  evaluating  the  platoon  Table 
IX  battle  rtn. 

Presents  representative  and  challenging  situations. 
Evaluation  based  on  a  variety  of  eeasires  and  scoring 
procedures. 


A- 4  5 


8m: 

ID  Nutter: 


Data: 

Title: 

Author: 

Corporate  Author: 


Keg  Points: 


121 

ARI-TR-457;  AD-A096  360 
flag  I960 

Methods  of  Evaluating  Tank  Platoon  Battle  Run  Performance 
Itwoton,  0.  A.,  Allen,  T.  U. ,  t  Johnson,  E. 

American  Institutes  for  Research 
Presents  recoeeended  tasks,  conditions,  constructs, 
measures  and  standard  that  increase  objectivity  shile 
retaining  diagnostic  capctoility 
Many  ‘standards'  are  in  fact  subtasks.  See  Itwaton, 
Fingerean,  Bogcan.  Score  aggregation  required  such  data 
froe  many  iterations  by  same  unit,  clearly  not  feasible. 
Arbitrary  standards  are  appealing.  Tng  Ngrs  need 
aggregation  for  diagnosis. 


Set- 

ID  M umber 
Date: 

Title: 

Author: 

Corporate  Author: 
8uemary: 


Key  Points: 


8m: 

10  Nwber: 

Date: 

Title: 


Author: 

Corporate  Author: 


Key  Points: 


122 

ARI-TR-374;  AD-A135  4B6 
Mar  1962 

Methods  of  Evaluating  Tank  Platoon  Battle  Run  Performance: 

A  Perspective 

Uheaton,  0.,  t  Bogcan,  0. 

AIR 

Examines  and  discusses  major  issues  that  have  significant 
(■plication  on:  method  of  main  gun  firing,  performance 
measurements,  and  purpose. 

Recommmend  use  of  additional  and  ieproved  evaluation  of 
tank  platoon  gunnery  and  tank  skills.  Presents  three 
options  for  measuring  platoon  performance. 


123 

Not  Available 
I960 

Military  Research  on  rerformance  Criteria:  A  Change  of 
Eephasis  (Article  appearing  in  Human  Factors  Journal,  vol 
22,  rr.  2> 

Uhl oner,  J.  E.,  8  Drucker,  A. 

Not  fcai table 

Points  out  change  of  emphasis  in  Military  research.  Trend 
is  amay  from  school  grades  and  subjective  roting  tomards 
performance  testing.  Recommends  use  of  situational 
performance  testing  (SQT,  real  train,  etc).  Recommends 
Systems  Measurement  Bed. 

Need  total  evaluation  of  unit,  team,  or  group  performance. 
Must  determine  systems  output  criterion  (i.e.  ARTEP). 
Systems  Measurement  Bed  deals  eith  overall  system's 
influence  on  individual's  performance  (Total  Mission 
Effect iveness). 


A-46 


SYNOPSES 


Seq: 

10  Number: 

Dot*: 

Till*: 


Author: 

Corporal*  Author: 
Stmmary: 


Key  Points: 


124 

AO-A112  937 
Doc  1961 

Multi-Attribute  Utility  Thoory  to  Assist  Top-Level 
Acquisition  Owcision-ftoung  (Master's  Thesis  free  Naval 
Post  Qraduate  School) 

Goren,  R. 

Naval  Post  Qraduata  School 

Search  for  decision-mcfcing  technique  that  can  best  serve 
top-level  acquisition  decision  Making.  Suggests  use  of 
flul  1 1 -Attribute  Utility  Theory. 

Presents  Methodology  and  effort  to  determine  key  decision 
Maker  information  needs. 


Seq: 

10  Number: 

Date: 

Title: 


Author: 

Corporate  Author: 
Suemary: 


Key  Points: 


125 

A0-P001  146 
Jan  1962 

Nee  tools  for  assessing  alrcraft/pi  lot  performance  (Paper 
presented  at  AIAA  workshop:  Flight  testing  to  identify 
pilot  workload  and  pilot  dynamics,  Edwards  AFB,  CA> 

York,  R.,  Montgomery,  L.,  S  Petro,  J. 

SRI  International 

Report  presented  at  AIAA  workshop:  f I  ight  testing  to 
identify  pilot  workload  and  pilot  dynamics.  Attempts  to 
assess  all  three  areas  of  aircraft  weapon  system 
effectiveness  (aircraft  performance,  pilot  physiological 
response,  and  armament  utilization). 

Recommends  multi-dimensional  approach  to  mtasirt  pilot 
workload,  stress,  and  performance  quantitatively.  Presents 
description  of  tactical  aircrew  combat  training  system 
(TACTS)  features. 


Seq: 

10  Nutfw: 

Date: 

Title: 

Author: 

Corporate  Author: 
Summary: 


126 

AlSO-tt-2285 
June  1961 

Notes  on  Human  Performance  Analysis 

Hoi  Inage  I ,  E. ,  Pendersen,  0.  M.,  t  Rasmussen,  J. 

Risoe  National  Lab 

Framework  for  the  integration  and  analysis  of  human 
performance  in  nuclear  environments.  Identifies  foir 
sources  of  data:  special  post  incident,  plcnt  interviews, 
training  simulators,  and  research  simulators. 

Provides  an  analysis  scheme  and  discusses  how  the  results 
from  the  different  levels  may  be  used  for  various  purposes. 


Key  Points: 


8VM0P8ES 


Seq. 

10  Hiabar: 

Data: 

Title: 

Author: 

Corporate  Author: 
Summary: 


Kay  Points: 


127 

Hot  Avail  Ala 
Hot  tool  I  Ala 

Optimization  of  Training  System* 

Cronhola,  J.  H. 

PH  TAHOE 

Looks  at  optiaizotlon  of  ski  1 1 -da fi nod  task  sequence  as  a 
aoans  to  training  systaa  optiaizotlon. 

Points  to  skill  defined  task  sequence  as  a  eethod  to 
optiaiza  training. 


Soa: 

ID  MuAer: 

Date: 

Title: 

Author: 

Corporate  Author: 
Suaaary: 

Key  Points: 


126 

HR I -AN-84 -47;  AD-A138  065 
FA  1684 

Performance  Feedback:  A  Review  of  its  Psychological  and 
Behavioral  Effects 

llgen,  0.,  Fisher,  C.,  I  Taylor,  H. 

Purdue  University 

Identifies  characteristics  of  feedback  Aich  aay  lead  to 
acre  effective  use  of  feedback. 

Points  out  the  i sport ance  of  understanding  shat  goes  on 
between  the  adeinistration  of  feedback  and  the  subject's 
selection  of  a  response. 


Seq: 

ID  HuAer: 

Oats: 

Title: 

Author: 

Corporate  Author: 
Siemary: 


Key  Points: 


129 

AD-A110  669 
Sept  1981 

Performance  Heasurement  and  the  Navy's  Tactical  Aircrew 
Training  Systaa  (TACTS) 

Stoffer,  0. 

Human  Factors  LA,  NTEC 

Describes  develops  art  and  use  of  Tactical  Training  System 
(TACTS)  as  a  aeons  for  training  advanced  air  coobat  skills. 

Indicates  current  limitations  in  Perforaance  Hsasvrement 
System. 

Cost  Effectiveness  Assessment  results  indicate  TACTS/ACHI 
reduces  training  costs  by  more  than  $100  mil  I  ion  annually. 
Presents  topics  to  consider  then  developing  Perforaance 
Heasurement  System. 


A- 4  8 


Seq:  130 

10  tfcAber:  Hot  Available 

Oat*.  1969 

Title:  Performance  Heasirement  System  for  Training  System 

Development  (Paper  presented  at  the  7th  l/ITEC,  Orlando, 

FL> 

Author:  Pmtt* t.  A.,  S  Hogruder,  P. 

Corporate  Author:  Hot  ffc*ai  table 

Summary:  Presents  integrated  approach  for  the  development, 

integration,  and  management  of  performance  measurement  in 
training  systems. 

Key  Points:  ffcprooch  organizes  and  translates  user  requirements  throutfn 

front-end  analyses  into  a  set  of  qualitative  aid 
quantitative  performance  measures. 


Seq:  131 

ID  Humber:  HfMC  2994 

Date:  1979 

Title:  Performance  Objectives  for  Infantry  8quad 

Author:  Hot  App 1 1 cable 

Corporate  Author:  US  Harine  Corps 

Summary:  Provides  performance  objectives  for  Infantry  squads 

preparing  for  HC  Combat  Readiness  Evaluation  8ystem 
(HCCHES).  Identifies  tasks,  level  (echelon),  conditions 
aid  requirements  (standards). 

Key  Points:  Looks  at  measurement  of  squad  performance  In  a  variety  of 

combat  situations. 


Seq:  132 

ID  Humber:  flBI-FH-81-27;  AD-A127  097 

Date:  July  1961 

Title:  Perspectives  on  Battalion  Training  Hanagement 

Author:  Hill,  H.,  6  Sticht,  T. 

Corporate  Author:  ARI 

Scmmary:  Four  battalion  commanders  mere  interviewed  to  explore  their 

philosophy  and  goals  for  training,  training  management 
practices,  and  training  and  evaluation  techniques. 

Key  Points:  Problems  found  with  SQT  tasting  and  flRTEP  and  a  less 

obvious  form  of  evaluation  that  is  often  used  by  bn  edrs  on 
a  dai ly  basis. 


A-49 


Seq: 

ID  Kobuv 

Data: 

Titla: 


Author: 

Corporata  Author: 
Siaacry: 


Kay  Points: 


133 

Not  Aval  labia 
Oct  1377 

Planning  for  Aircraa  Parfomanet  Heasureoent  AU)  (Popar 
pram  tad  at  tha  Productivity  Erhoncaoant:  Personnel 
Parforaanca  Assassaant  In  Novy  Systaas  S^posiue,  San 
Diego,  CA> 

Uoog,  U.  t  Knoop,  P. 

Not  Auoi labia 

Suaaarizas  AFHRL  effort*  of  davaloping  aaaauras  of  aircraa 
proficiency  ahlch  could  be  used  in  both  ground-based  and 
airborne  anvironaant*.  Points  to  uresolved  problaa  of 
aaasuraaant  systaa  validation. 

Presents  efforts  of  developing  objective  parforaanca 
eeasiraaant  ahich  could  lead  to  the  developeent  of 
operational  airborne  aaasuraaant  systaes. 


Seq: 

ID  Nu*er: 

Data: 

Titla: 

Author: 

Corporata  Author: 
Suaaary: 


Kay  Points: 


134 

18K10-0SUT 
Jui  85 

POI,  H1/N1A1  Abroas  fraor  Craaaon  (Draft) 

Not  Applicable 
US  fray  fraor  School 

Outlines  tha  course  of  instruction  to  train  soldiers  to 
function  as  SL1  creeaan  on  an  HI  Abraas  tank  NOS  trained, 
19K10.  Provides  scope  (tasks)  of  each  period  of 
instruction,  end  ahethor  tasks  are  'taught  to  standard*. 


Seq: 

ID  Nusber: 

Date: 

Title: 

Author: 

Corporata  Author: 
Staaary: 


Kay  Points: 


135 

19010-0SUT  (H1 13) 

Hot  1085 

POI,  H1 13  Cavalry  Scout  fray 
Not  Applicable 
US  Aray  fraor  8chool 

Outlines  tha  course  of  Instruction  to  train  soldiers  to 
function  as  8L1  cavalry  scouts  on  an  Hi  13.  HOS  trained, 
10010.  Provides  scope  (tasks)  of  each  period  of 
Instruction,  and  ahether  tasks  ere  ’taught  to  standard*. 
Provides  basis  for  analysing  corns*  content. 


0 


I 


-5 


Stq: 

ID 
tote: 

Titte: 

Author: 

Corporate  Author: 


Kag  Points: 


136 

POJ  19010-08UT  CH3> 
tpr  1965 

POI,  K3  Brad ley /CPU  Scout  (Draft) 

Not  ftopllcahla 
US  Amy  Amor  School 

Outlines  the  course  of  instruction  to  train  soldiers  to 
function  as  8L1  cavalry  scouts  on  a  CPU.  NOS  trained, 
19010.  Provides  scope  (tasks)  of  each  period  of 
instruction,  and  ehether  tasks  are  'taught  to  standard'. 
Provides  basis  for  amlyzing  corse  content. 


8eq: 

ID  toMbr  : 

Date: 

Title: 

Author: 

Corporate  Author: 

Sueeary: 


Keg  Points: 


137 

19E10-0SUT 
Jun  1965 

POI,  NB0A1  Armor  Creep  an  (Draft) 

Not  ffplicable 
U8  Amy  Amor  School 

Outlines  the  corse  of  instruction  to  train  soldiers  to 
function  as  SL1  tank  creeean  on  on  N60A1  tank.  NOS 
trained,  19E10.  Provides  scope  (tasks)  of  each  period  of 
instruction,  and  ehether  tasks  are  'tau^tt  to  standard*. 
Provides  basis  for  analyzing  corse  content. 


Seq: 

ID  Nufcer: 


Oate: 

Title: 

Author: 

Corporate  Author: 


KOy  Points: 


136 

19E10-0SUT-ASIB8 
Jun  1985 

POI,  N60R3  Armor  Creeean  (Draft) 

Not  Applicable 
U8  Amy  Amor  School 

Outlines  the  corse  of  instruction  to  train  soldiers  to 
function  as  SL1  tank  creeean  on  an  N60R3  tank.  HOS 
trained,  19E10;  ASI68  aearded  to  NOS  19E.  Provides  scope 
(tasks)  of  each  period  of  instruction,  and  ehether  tasks 
are  ‘taught  to  standard'. 

Provides  basis  for  analyzing  corse  content. 


A-51 


IWOPSD 

*•9: 

ID  Nimbor: 

Doit: 

Title: 


Author: 

Corporate  Author: 


I Coy  Points: 


190 

Not  Available 
Now  1060 

Policy  laplicatlons  toalysis:  A  Nathod  for  laprowing 
Policy  Aasoarch  and  Evaluation  (Apposing  in  Book  laprowing 
Educational  Evaluation  flatted*:  lapoct  on  Policy  -  Editor: 
Carol  8.  A* I on i ana) 
fledey,  D. ,  0  Sterner,  A.  J. 

NTS  Aasoarch  Corporation 

Dtscribos  the  development  and  application  of  Policy 
laplicatlons  Analysis;  designed  to  help  people  eho  are 
planning  or  conducting  evaluations  to  tailor  their 
information  so  that  it  has  optieal  potential  for  being  used 
and  acted  tpon. 

Eephasizes  need  to  start  evaluation  planning  si  th  a  careful 
asses  seen  t  of  policy  eakers  information  needs.  Builds  i^on 
Delphi  flatted  aid  Scenario  Uriting. 


Seq: 

ID  Number: 

Date: 

Title: 

Author: 

Corporate  Author: 
Si  merry: 


Key  Points: 


140 

HPAOC-Sfl-62-18;  AD-A113  491 
Her  1962 

Potential  Applications  of  Coeputer  Assisted  Instruction  to 
P-3  Rircres  Trainer 

Acrks,  L-  J.,  Hoskins,  U.  W.,  I  Kribs,  H.  D. 

Instructional  Science  t  Development 
Exaeines  P-3  aircree  training  syllabus  to  find  out  hoe 
training  effectiveness  could  increase  by  a  shift  to  CAI. 
Presents  analysis  and  comparison  of  various  CAI  systems  and 
the  current  P-3  media  mix  to  determine  the  most  effective 
training  approach. 

Identifies  areas  in  curriculum  shore  CAI  might  improve 
effectiveness  of  training,  flethodology  could  be  of 
interest. 


Seq: 

ID  timber: 

Date: 

Title: 

Author: 

Corporate  Author: 
Sumeary: 


141 

ARl-TR-813 
Apri I  1962 

Prediction  of  Training  Device  Effectiveness:  A  Revies  of 
Army  Node I* 

Tufano,  D.  A.,  t  Evans,  A.  A. 

Not  Available 

Looks  at  four  training  device  effectiveness  predictive 
models,  knoon  as  TRAIN  UICE. 

Looks  at  Training  Effectiveness  models. 


Key  Points: 


8VM0P8E8 

8m: 

10  Nuber: 

Data: 

Title: 

Author: 

Corporate  Author: 
Sueeary: 


Kay  Points: 


142 

Not  Aval labia 
Not  Aval  labia 

Aaliabia  Noasuraoant  of  Task  ParforMnea  Froa  Training  to 
tho  Job 
Osborn.  U. 

HuaARO 

Focusas  on  sourcas  of  enrol  iebi  I  ity  in  testing  soldier 
porforaanca  at  tho  coaptation  of  training  and  again  after 
job  ossigneent. 

Prasants  sourcas  of  test  reliability. 


Sm* 

10  Nutoor: 

Oata: 

Titia: 

Author: 

Corporate  Author: 
Sueoary: 


Kay  Points: 


143 

flRI-AH-77-7;  AD-A077  027 
Oac  1077 

Aaport  of  Exercise  Observations:  Oparational  Rood  inass 
Training  Tost  CORTT ) 

Ryan,  T.  0..  t  Votes.  L.  0. 

ARi 

Prasants  evaluation  activity  and  racoaaandotions  of  tha  ARi 
fiald  unit  USAREUR  concerning  battalion  ORTT. 

So  Melted  perceptions  and  suggestions  for  ioproving  the 
ORTT  froa  unit  personnel,  host  relevant  recoaaendotlon  is 
the  need  for  standardized  porforaanca  feedback. 


Saq: 

ID  Nuebar: 

Date: 

Title: 

Author: 

Corporate  Author: 
Suonary: 


Kay  Points: 


(44 

ARI-TR-78-A18;  AD-A056  054 
Jun  1078 

Research  on  Training  for  Brigade  Coaaand  Groins:  Factors 

Contributing  to  Uni t  Coabat  Readiness 

Olastead.  J.  A..  Baranick,  H.  J.,  8  Elder,  B.  L. 

Systea  Bevel opaent  Corp 

Exaainas  relationship  bateaen  brigade  coaaand  group 
effectiveness  and  porforaanca  of  ARTEP  during  Coeputer 
Assisted  Han  Naneuver  Exercise  <CA(t1S>. 

CflfftS  is  a  too-sided  battle  sieuiation.  Coabat  outcoaes 
detarained  by  coaputer  ohich  provides  rapid  calculation  and 
feedback  of  engogeaent  results  betoeen  friendly  and  throat 
forces.  Can  provide  end-of-exercise  r parry  of  status  of 
both  forces. 


A-53 


8VN0P8E8 


Seq: 

ID  feet**: 

Dot*: 

Title: 

Author: 

Corporal*  Author: 
Sueaary: 


K*g  Points: 


145 

NTEC  83-C-0015-1;  AD-A154  400 
har  1005 

Results  of  th*  (-art  Task  Ship  Handling  Trainer 
Pre-prototype  Training  Effectiveness  Evaluation  (TEE) 
Hanley,  fl.  L. 

Ship  toalytics,  Inc 

Looks  at  PARTSHIP  (part  task  trainer  for  ship  Fondling). 
Presents  TEE  ahich  eas  conducted  on  too  areas  of  ship 
handling.  Perfores  pretest- training-post  test  on  training 
device  and  coepares  against  full  bridge  simulator. 
Eephasizes  problems  in  sieulation  training  research  due  to 
the  iswvoi labi  1  ity  of  criterion  eeasures  of  caeplex, 
real-eorld  perforeance. 


Seq: 

ID  Huaber: 

Date: 

Title: 

Author: 

Corporate  Author: 
SuMMry: 


Key  Points: 


146 

ARI-flft-1329;  PD-A130  071 
flay  1961 

Revise  of  Methodologies  for  Analysis  of  Collective  Tasks 
Bauer,  R. 

RRI 

Presents  state  of  the  art  eethodoiogies  for  analysis  of 
collective  tasks. 

Cooper i son  of  different  eethodoiogies  indicated  sao*  coeeon 
technique  but  little  agreement  on  basic  concepts, 
terminology,  or  even  products  of  collective  analysis. 


Seq: 

ID  fester: 

Date: 

Title: 

Author: 

Corporate  Author: 
Sueeary: 

Key  Points: 


147 

AD-P002  317 
Hot  Available 

Risk  Hanogeeent  in  a  Hultiobjective  Decision-feeing 

Fraeoeork 

Haiees,  V.  V. 

Systees  Engineering  Departeent 

Looks  at  risk  assesseent  os  an  integral  part  of  th* 

decision-ocking  process. 

Effect  of  risx  eanageeent  on  decision  eaking. 


A-54 


SVMOPSES 


Saq: 

ID 

Data: 

Tltla: 

Author: 

Corporota  Author: 
Sueeary: 


Kay  Points: 


148 

RD-B955  30QL 
1988 

Role  of  Oparotions  Research  In  miliary  Decision-MUng 
Anon,  R.  H. 

US  Rray  C08C 

Provides  allltory  dacision  arftar  aith  Insist  Into  Opns  Ras 
to  aid  in  objectively  structuring  problaas. 

Prow i das  aeons  to  ieprove  quantitative  procaduras  as 
Justification  to  dacision  aakars  decisions  and 
ncoMinggiiofii, 


Saq: 

10  Nurtw*: 

Data: 

Title: 

Author: 

Corporate  Author: 
Sueeary: 


Key  Points: 


140 

RR I -flH-78- 12;  RD-A077  981 
Apr  1978 

Score  Quality  Issues  Related  to  Individual  and  Ueapon  Creo 
Criterion  Reference  Perforeance  Tests 
Steinheiser,  F.,  t  Snyder,  C.  U. 

RRI 

Presents  issues  related  to  cr  I  ter  ion-par  forwnce  tasting 
ehich  should  be  considered  ehen  developing  individual  and 
eeapon  cree  tests. 

Pass/Fail  decisions  oust  be  node  free  o  fairly  seal  I  staple 
of  I  teas,  therefore  errors  ora  unavoidable. 


Saq: 

ID  Nuebar: 

Date: 

Title: 

Author: 

Corporate  Author: 
Suae  cry: 

Kay  Points: 


150 

NUREU/CB-3728 
hoy  1984 

Sieulator  Fidel ity  and  Training  Effectiveness:  A 
Coeprehensive  Bibliography  elth  Selected  Amo  tot  ions 
Bolton,  P.,  Faigenblue,  J. ,  Hope,  A.  8  Rankin,  U. 
hot  Avai table 

finnoted  bibliography  of  sieulator  fidelity  and  training 
effectiveness  studies. 

Identifies  several  "perforeance  eeasureaent"  articles 
(training  device  related). 


A- 5  5 


8VN0PSES 


Seq: 

(0  Huaber: 

Oat*. 

Title: 

Author: 

Corporal*  Author: 
Summary: 


151 

flFHAL-TA-60-63 
Jan  1962 

SI  aula  tor  Training  Requirements  end  Effectiveness  Study 
<STRES):  Executive  fhennrj 
Seep  I  a,  C.  A. 

Canyon  Anarch  Group,  Inc.  _ 

On*  of  ?  technical  reports  prepared  for  STAES.  SuMarizas 
contents  of  other  six  reports  ahich  address  aircre* 
training  device  Issues  to  include:  fidelity,  instructional 
support  features,  and  utilization  in  aircre*  training 


Key  Points:  Refers  to  :  Instructional  Features  Uoluee  ,  Chapter  IU 

(Monitor  and  Evaluate  Performance)  for  information  on  hoe 
performance  is  monitored  «td  evaluated. 


Seq: 

ID  Number: 


Date: 

Title: 

Author: 

Corporate  Author: 


Key  Points: 


152 

AD-A043  543 
June  1977 

Social  Structure  of  Decision  flaking 
Vrooe,  U.  H. 

Vale  University 

Overvie*  of  research  conducted  under  the  areas  of  : 
leadership  styles,  leadership  development, training,  etc. 
Effect  of  leadership  styles,  perceptions,  and  structure  on 
decision  mcfclng. 


Seq: 

10  Number: 

Date: 

Title: 

Author: 

Corporate  Author: 
Summary: 


Kay  Points: 


153 

AD-A096  202 
Feb  1961 

Soldier  Capability  -  Army  Combat  Effectiveness  (SCACE) 
Volume  II  Selected  Bibliography 
Toamepuu,  J. 

USA  Soldier  Support  Center 

Study  undertaken  to  quantify  the  relationship  beteeen  the 
capability  of  soldiers  and  the  coabat  effectiveness  of 
eeqpons,  units  and  forces.  Presents  selected  bibliography. 
Literature  revise  reiterates  that  soldier  capdbi I ities  are 
a  major  determinant  of  combat  effectiveness.  Points  out 
that  variables  used  to  deters  in*  capabilities  are 
identifiable,  eeasurable  and  useful  for  prediction. 


A-56 


10 
Data: 

Title: 

Author: 

Corpora  to  Author: 


Kay  Points: 


OTP  D-63E39-SH-T0 
Apr  1005 

Soldiar's  tanuol  Tralnor's  Guide 
Not  Applicable 
U8  Aray  Ordnanco  School 

Contains  standardized  training  objectives,  in  fora  of  task 
n— nr  las,  usad  to  train /ova  I  into  on  critical  tasks  ohich 
SMJport  init  aissions  dir ing  sort! at. 

This  par  to  ins  to  skill  level  3,  4,  and  9,  ohich  is  outsido 
tho  fVtt  study  effort. 


Sag: 

10  Nuabar: 

Data: 

Title: 

Author: 

Corporate  Author: 
Suawry: 


Kay  Points: 


1S9 

STP  Q-63E12-SM 
Apr  \9BS 

Soldiar’s  Hanual,  HI  Abraes  Tank  Systea  Hachanic 
Not  APPl  iocA><* 

US  Arey  Ord’wnca  School 

Provides  standardized  training  objectives  in  fore  of  task 
sunearies  usad  to  troin/avoluota  soldiers  on  critical  tasks 
supporting  ini  t  aissions  ckrtng  oar  ties. 

Provides  basis  for  analyzing  HOS-related  job  requireeants. 


8oq: 

ID  Nuabar: 


Date: 

Title: 

Author: 

Corporate  Author: 


Kay  Points: 


196 

STP  P-63N12-SN 
Nov  1069 

Soldiar's  Hanna I ,  H60A1/A3  Tank  Systaa  Hachanic 
Not  App> (cable 
US  fray  Ordnance  School 

Provides  standardized  training  objectives  in  fore  of  task 
si— pries  used  to  train /evaluate  soldiers  on  critical  tasks 
supporting  unit  aissions  dir  ing  aartiaa. 

Provides  basis  for  analyzing  HOS-related  job  requireeants. 


A-57 


A-58 


8VN0PSE8 


8m: 

ID  Nufcer: 

Data: 

Tills: 

Author: 

Corporate  Author: 
Summary: 


Key  Points: 


160 

AD-A001  401 
Hot  Available 

Success  and  Failure  in  Skill  Qualification  Totting  &  Troop 

Uioms 

tier  wan,  j. 

ARI 

Explore*  roosons  for  succoss  and  failire  in  SQT  lotting. 
Usos  on  I  it  tod  soldiers  oho  toko  SQTt  at  tourco  of 
information. 

Results  indicate  that  soldiers  oophatizo  the  ioportanco  of 
performing  tasks  as  part  of  unit  duty  in  order  to  bo 
prepared  to  anioor  questions  in  the  skill  cooponont  portion 
of  the  SQT.  Discusses  SQT  and  Its  cooponents. 


Sm: 

ID  Humber: 

Date: 

Title: 

Author: 

Corporate  Author: 
Stamary: 

Key  Points: 


161 

RRI-RP-70- 15;  AD-A082  706 
Nov  1070 

Tank  Cree  <H60A1)  Performance  Exercise 
O'Brien,  R.,  Harris,  J.,  6  Osbum,  U. 

HueRRO 

Provides  test  exercises  and  administrative  guidance  for 
evaluating  the  readiness  of  H60RI  tank  crews. 

Provides  integrated  train-ip  package  for  annual  ginnery 
evaluation.  Individual  and  team  tasks  are  measured  via 
rating  sheets  and  scorecards  <go/no  go). 


8eq: 

ID  Number: 

Date: 

Title: 

Author: 

Corporate  Author: 
Summary: 


162 

RRI-RP-70-13;  AD-R082  560 
Nov  1070 

Tank  Crewmen  (N60R1)  Readiness  Tests 
O'Brien,  R.,  Harris,  J.,  t  Osbum,  U. 

HumARO 

Provides  tests  and  administrative  guidance  for  evaluating 
H60R!  tank  crewmen  job  readiness.  Tests  cover  knowledge  and 
skill  aspects  of  those  tasks  that  are  most  relevant  to 
crew  gunnery  proficiency. 

Presents  two  types  of  readiness  tests:  written  and  hands  on 
tests  which  provide  commander  with  diagnostic  tool  for 
determ  ining  performance .  Utilizes  go/no  go  evaluation. 


Kay  Points: 


8VTCPSES 

Saq: 

ID  H daber: 

Data: 

Title: 

Author: 

Corpora ta  Author: 
Sueaary: 


Kag  Points: 


163 

ARI-AN-85-12;  AD-A149  662 
Jan  1985 

Taaa  Diaansions:  their  identity,  thair  Measurements,  and 
thair  Raiationships 

Mleva,  U.  F.,  Fleishman,  E.  A.  6  Alack,  A. 

Response  Analysis  Corporation 

Presents  initial  phasa  of  affort  a  load  at  answering  basic 
questions  about  the  nature  of  taaa  parforaance  and  factors 
affecting  it.  Proposes  taxonoay  of  taaa  parforaance 
diaansions. 

Methodologies  are  needed  to  evaluate  togas  along  identified 
teas  parforaance  dioension.  Several  hove  reached  a  stage 
of  deveiopeent  ehich  can  be  applied  to  group  parforaance: 
binary  decision  floe  diagrams,  BARS,  t  profile  analytic 
aethods. 


Sm: 

ID  Nuaber: 

Data: 

Title: 

Author: 

Corporate  Author: 
6uMMry: 

Key  Points: 


164 

S8I-AD-E751-034;  AD-8067  734L 
Jun  1982 

Training  Aspect  of  Aeserve  Battalion  Coabat  Aeadiness 
(Master's  thesis) 

Ash,  S.  E. 

USAray  COSC 

Study  of  the  training  variable  of  the  coabat  read  Inass 
equation  as  It  concerns  AC  eanouver  battalions. 

Study  concludes  that  readiness  determination  Is  haepered  by 
a  lack  of  a  eel ! -defined,  aeasurable  definition  of 
readiness. 


Saq: 

ID  Nuaber: 

Dote: 

Title: 

Author: 

Corporate  Author: 
Sueaary: 


Key  Points: 


165 

AR I -TP-376;  AD-A075  414 
Jun  1979 

Training  Battalion  Comaand  Groups  in  Si  aula  ted  Coabat: 
Identification  and  Measurement  of  Critical  Performances. 
Kaplcrt,  I.  T.,  t  Barber,  H.  P. 

ARI 

Investigates  behavior  of  23  battalion  coaeand  p-otps  in  a 
siaulated  coabat  environment.  Significant  differences 
found  among  ratings  of  the  same  command  group  by  several 
observers. 

Results  indicate  need  to  develop  more  objective  measures  of 
performance  and  identify  those  subtasks  for  ehich  different 
raters'  perspectives  should  produce  valid  differences  in 
ratings. 


A- 60 


Ssq:  166 

10  * mb*'.  RRI-flR-1291;  AD-A122  777 

Date:  Sap  1080 

Till*:  Training  Davies  Effectiveness:  Formulation  and  Evaluation 

of  a  Itathodology 

Author:  Bickloy,  M.  R. 

Corporate  Author:  ARI 

S— sary:  Presents  modal  to  da  lamina  training  device  effectiveness. 

flodal  cons  i  dors  only  antaeadant  si  aula  tor  training. 

Kay  Points:  nodal  not  eoneamad  oith  fidality  and  mot  Isa  but  dimctly 

oddrassa s  effectiveness  of  tha  si  aula  tor  In  docmasing 
raquirad  aircraft  training  tiaa. 


Smq: 

167 

10  Number: 

flRI-flFh  13676;  fiO-A158  018 

Oata: 

Fab  1884 

Titla: 

Training  Effaetiuanass  ftoalysis:  8tatus  of  Institutional 
and  Uhit  Mortar  Training 

Author: 

Fusha,  J.,  Pam,  A..,  6  Thompson,  T. 

Corporate  Author: 

Litton 

Summery: 

Looks  at  Institutional  and  unit  training  programs,  prablaas 
and  dsficiancias  and  provides  raco— andatlons  for 
iaprouaaant.  Extensive  analysis  of  POI,  unit  training 
programs,  flRTEP,  and  liwa  firm  oxorclsms. 

Major  problems  Idantifiad  mam:  no  sal  act  I  on  criteria  for 

11C  mortar  era— an,  no  institutional  training  for  8L  2, 
ate. 

Kay  Points: 

Saq: 

168 

10  Nuabar: 

ARI-flN-82-27;  RD-A140  887 

Oata: 

Aug  1082 

Titla: 

Training  Effaetiuanass  as  a  Function  of  Training  Davies 

Fidality 

Author: 

Baum,  0.  t  Riadal,  8. 

Corporate  Author: 

Honay—ll  Systems  and  Aassarch  Ctr 

Su—ary: 

Study  to  da  lamina  tha  offsets  of  rsducsd  training  device 
fidality  on  looming  and  psrforsancs  of  a  perceptual motor 
maintanancs  task  (bicyels  ah— 1  training). 

Kay  Points: 

Looks  at  offset  of  training  doviea  fidality  on  perceptual 
task  parforaancs . 

A-61 


Saq 
ID 
Dot*: 

Title: 

Author: 

Corporota  Author: 


Kay  Points: 


1M 

AAI-TPM42;  AD-A126  m 
Fab  IMS 

Training  Effectiveness  Evaluation  (TEE)  of  tha  Advanced 
Fira  Fighting  Training  System. 

Cordell,  C.,  Nuttar,  A.,  I  HcOanlel,  U. 

Training  Analysis  t  Evaluation  Group 

Training  Effaetivanass  Evaluation  of  tha  Navy  Advoncad  Fira 
Fighting  Training  dystaa.  Found  that  firas  ora 
suffielantly  raalistle  that  positiva  training  did  occur. 
Lookad  at  9  ooaponants  of  training  systaa:  publications, 
Oavlca  19F1,  structure,  supporting  subsystems,  and 
curriculun.  Uses  aoplrlcal  noncoaparotiva  evaluation 
tachnlque.  Control  group  ■  course  objectives. 


Saq: 

170 

10  Nuabar : 

TRASANR-TR-49-82;  AD-A122  TOG 

Data: 

Oct  1982 

Titla: 

Training  Effactiuanasss  too  lysis  -  A  Process  in  Evolution 

Author: 

Miller,  C.,  t  Southard,  L. 

Corporate  Author: 

USflR  TRA00C  Systems  Anal.  Activity 

Summary: 

Provides  brief  history  of  Training  Effaetivanass  too  lysis 
(TEA)  Division  in  TAASAHA.  Suamarizas  tha  cost  analyses  in 

TEAS. 

Kay  Points: 

Summarises  TEA  studies  to  include:  Evaluation  of  training, 
training  devices,  soldier /herdaara  interface,  and  N08 
selection  criteria.  Ulthln  evaluation  of  training  looks  at 
training  of  nee  equipment,  Institutional  tmg,  and  Unit 
tmg. 

Seq: 

171 

10  Nuabar: 

ARI-AR-1196;  AD-A190  632 

Oata: 

Jut  1989 

Titla: 

Training  Extension  Course  Research:  Aeviao  of  tha 
Litarature  on  Cost  and  Training  Effaetivanass 

Author: 

Sasona,  P.  G. 

Corporate  Author: 

LI  t  ton-tie  1  Ionics 

Summary: 

Aeviao  of  1  Itarature  on  cost  and  training  effaetivanass. 
Attempts  to  develop  a  CTEA  for  the  Army’s  TEC. 

Kay  Points: 

Aaports  on  ailitory  and  non-mi  1  Itary  literature  dealing 
eith  state  of  the  art  eethods  and  techniques  applicable  to 
the  performance  of  CTEA. 

A-62 


•nq: 

10 

Data: 

Title: 

Author: 

Corporota  Author: 
ftaaory: 


Kay  Paints: 


172 

TR  606 
Jen  1064 

Transfer  of  Troining:  An  Interpretive  Rev  lee 
Com  tor,  8.  H. 

PR  I 

Study  looks  at  inforaation  processing  and  eeaory  process  as 
providing  a  basis  for  explaining  and  predicting  transfer  of 
training  effects. 

Looks  at  autoeatized  performance  as  an  iaportant  factor  to 
explain  and  predict  transfer  of  training  effects. 


Seq: 

10  Nutfw: 

Dote: 

Title: 


Author: 

Corporate  Author: 


173 

flfll-TR-491;  AD-A135  450 
Sep  1072 

Uork  Envlronaent  Questionnaires  and  Arey  Unit  Effectiveness 

and  Satisfaction  Measures 

Spencer,  L.,  Kleep,  0.  S  Cullen,  6. 

Not  Aval  table 

Rev  lees  existing  ailltary  and  civilian  eork  environaents 
and  organizational  cl  I eats  questionnaires.  Identifies 
eepirical  aeasures  of  Arey  unit  effectiveness. 

Looks  at  inspection  reports,  elssion  accoopl  Isheent 
results,  efficiency  aeastres,  etc. 


Key  Points: 


